Ask General Programming Questions Not Worth Their Own Thread

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›9 »

tef: May 30, 2004; -> some l-system crap ->

REST isn't a toolkit, a library, or anything resembling a protocol. It's the name for opening a tab in a new window, It's a set of constraints the web browser put upon the design of HTTP.

It's literally about web browsing. If I can't convince you that the notion of "using one client for a variety of services, using self descriptive formats to expose a rich interface over a simpler rpc interface that supports middleware" is useful, what the hell do you do in the web browser anyway?

# ¿ Nov 9, 2018 23:53

Adbot: ADBOT LOVES YOU

# ¿ May 14, 2024 01:22

tef: May 30, 2004; -> some l-system crap ->

TheFluff posted:

people don't want to use one client when programming, tef. they want to go on npm and try to figure out which one of ten libraries that do almost the same thing suits their taste best

I know i'm old and out of touch but do you really have to put the boot in? Oh well.

# ¿ Nov 10, 2018 00:09

tef: May 30, 2004; -> some l-system crap ->

22 Eargesplitten posted:

Can someone explain in idiot terms what RESTful means?

Here's a long post that tries to explain what the hell is REST, anyway.

Originating in a thesis, REST is an attempt to explain what makes the browser distinct from other networked applications.

You might be able to imagine a few reasons why: there's tabs, there's a back button too, but what makes the browser unique is that a browser can be used to check email, without knowing anything about POP3 or IMAP.

Although every piece of software inevitably grows to check email, the browser is unique in the ability to work with lots of different services without configuration�this is what REST is all about.

HTML only has links and forms, but it's enough to build incredibly complex applications. HTTP only has GET and POST, but that's enough to know when to cache things, HTTP uses URLs, so it's easy to route messages to different places too.

Unlike almost every other networked application, the browser is remarkably interoperable. The thesis was an attempt to explain how that came to be, and called the resulting style REST.

REST is about having a way to describe services (HTML), to identify them (URLs), and to talk to them (HTTP), where you can cache, proxy, or reroute messages, and break up large or long requests into smaller interlinked ones too.

How REST does this isn't exactly clear.

The thesis breaks down the design of the web into a number of constraints�Client-Server, Stateless, Caching, Uniformity, Layering, and Code-on-Demand�but it is all to easy to follow them and end up with something that can't be used in a browser.

REST without a browser means little more than "I have no idea what I am doing, but I think it is better than what you are doing.", or worse "We made our API look like a database table, we don't know why". Instead of interoperable tools, we have arguments about PUT or POST, endless debates over how a URL should look, and somehow always end up with a CRUD API and absolutely no browsing.

There are some examples of browsers that don't use HTML, but many of these HTML replacements are for describing collections, and as a result most of the browsers resemble file browsing more than web browsing. It's not to say you need a back and a next button, but it should be possible for one program to work with a variety of services.

For an RPC service you might think about a `curl` like tool for sending requests to a service:

code:

$ rpctl [url]http://service/[/url] describe MyService
methods: ...., my_method

$ rpctl [url]http://service/[/url] describe MyService.my_method
arguments: name, age

$ rpctl [url]http://service/[/url] call MyService.my_method --name="james" --age=31
Result:
   message: "Hello, James!"

You can also imagine a single command line tool for a databases that might resemble `kubectl`:

code:

$ dbctl [url]http://service/[/url] list ModelName --where-age=23
$ dbctl [url]http://service/[/url] create ModelName --name=Sam --age=23
$ ...

Now imagine using the same command line tool for both, and using the same command line tool for _every_ service�that's the point of REST. Almost.

code:

$ apictl call MyService:my_method --arg=...
$ apictl delete MyModel --where-arg=...
$ apictl tail MyContainers:logs --where ...
$ apictl help MyService

You could implement a command line tool like this without going through the hassle of reading a thesis. You could download a schema in advance, or load it at runtime, and use to create requests and parse responses.

REST is quite a bit more than being able to reflect, or describe a service at runtime. The REST constraints require using a common fomat for the contents of messages so that the command line tool doesnt need configuring, require sending the messages in a way that allows you to proxy, cache, or reroute them without fully understanding their contents.

REST is also a way to break apart long or large messages up into smaller ones linked together�something far moe than just learning what commands can be sent at runtime, but allowing a response to explain how to fetch the next part in sequence.

To demonstrate, take an RPC service with a long running method call:

code:

class MyService(Service):
    @rpc()
    def long_running_call(self, args: str) -> bool:
        id = third_party.start_process(args)
        while third_party.wait(id):
            pass
        return third_party.is_success(id)

When a response is too big, you have to break it down into smaller responses. When a method is slow, you have to break it down into one method to start the process, and another method to check if it's finished.

code:

class MyService(Service):
    @rpc()
    def start_long_running_call(self, args: str) -> str:
         ...
    @rpc()
    def wait_for_long_running_call(self, key: str) -> bool:
         ...

In some frameworks you can use a streaming API instead, but replacing streaming involves adding heartbeat messages, timeouts, and recovery, so many developers opt for polling instead, breaking the single request into two.

Both approaches require changing the client and the server code, and if another method needs breaking up you have to change all of the code again.

REST offers a different approach. We return a response that describes how to fetch another request, much like a HTTP redirect. In a client library, you could imagine handling these responses, much like an HTTP client handles redirects too.

code:

def long_running_call(self, args: str) -> bool:
    key = third_party.start_process(args)
    return Future("MyService.wait_for_long_running_call", {"key":key})

def wait_for_long_running_call(self, key: str) -> bool:
    if not third_party.wait(key):
        return third_party.is_success(key)
    else:
        return Future("MyService.wait_for_long_running_call", {"key":key})

code:

def fetch(request):
   response = make_api_call(request)
   while response.kind == 'Future':
       request = make_next_request(response.method_name, response.args)
       response = make_api_call(request)

For the more typed among you, please ignore the type annotations, pretend I called `raise Future(....)`, or imagine a `Result<bool>` that's a supertype. For the more operations minded, imagine I call `time.sleep()` inside the client, and maybe imagine the Future response has a duration inside.

The point is that by allowing a response to describe the next request in sequence, we've skipped over the problems of the other two approaches�we only need to implement the code once in the client. When a different method needs breaking up, you can return a `Future` and get on with your life.

In some ways it's as if you're returning a callback to the client, something the client knows how to run to produce a request. With `Future` objects, it's more like returning values for a template. This approach works for paginating too�breaking up a large response into smaller ones.

Pagination often looks something like this in an RPC system:

code:

cursor = rpc.open_cursor()
output = []
while cursor:
    output.append(cursor.values)
    cursor = rpc.move_cursor(cursor.id)

Or something like this:

code:

start = 0
output = []
while True:
    out = rpc.get_values(start, batch=30)
    output.append(out)
    start += len(out)
    if len(out) < 30:
        break

Iterating through a set of responses means keeping track of how far you've gotten so far. The first pagination example stores state on the server, and gives the client an Id to use in subsequent requests. The second pagination example stores state on the client, and constructs the correct request to make from the state.

Like before, REST offers a third approach. The server can return a `Cursor` response, much like a `Future`, with a set of values and a request message to send for the next chunk, and the client pages through the responses to build a list of values:

code:

cursor = rpc.get_values()
output = []
while cursor:
    output.append(cursor.values)
    cursor = cursor.move_next()

code:

class ValueService(Service):
    @rpc()
    def get_values(self):
        return Cursor("ValueService.get_cursor", {"start":0, "batch":30}, [])

    @rpc
    def get_cursor(start, batch):
        ...
        return Cursor("ValueService.get_cursor", {"start":start, "batch":batch}, values)

The REST approach offers something different. The state is created on the server, sent back to the client, and then sent back to the server. If a Server wants to, it can return a `Cursor` with a smaller set of values, and the client will just make more requests to get all of them.

`Future` and `Cursor` aren't the only kind we can parameterise�a `Service` can contain state to pass into methods, too.

To demonstrate why, imagine some worker that connects to a service, processes work, and uploads the results. The first attempt at server code might look like this:

code:

class WorkerApi(Service):
    def register_worker(self, name: str) -> str
        ...
   def lock_queue(self, worker_id:str, queue_name: str) -> str:
        ...
   def take_from_queue(self, worker_id: str, queue_name, queue_lock: str):
       ...
   def upload_result(self, worker_id, queue_name, queue_lock, next, result):
       ...
   def unlock_queue(self, worker_id, queue_name, queue_lock):
       ...
   def exit_wotker(self, worker_id):
       ...

Unfortunately, the client code looks much nastier:

code:

worker_id = rpc.register_worker(my_name)
lock = rpc.lock_queue(worker_id, queue_name)
while True:
    next = rpc.take_from_queue(worker_id, queue_name, lock)
    if next:
        result = process(next)
        rpc.upload_result(worker_id, queue_name, lock, next, result)
    else:
        break
rpc.unlock_queue(worker_id, queue_name, lock)
rpc.exit_worker(worker_id)

Each method requires a handful of parameters, relating to the current session open with the service. What we'd rather use is some API where the state between requests is handled for us:

code:

lease = rpc.register_worker(my_name)

queue = lease.lock_queue(queue_name)

while True:
    next = queue.take_next() 
    if next:
        next.upload_result(process(next))
    else:
        break
queue.unlock()
lease.expire()

The traditional way to achieve this is to build these wrappers by hand�creating special code on the client to wrap the responses, and call the right methods. If we can link together a large response, we should be able to link together the requests, and pass the state from one to the next just like a `Cursor` does.

Instead of one service, we now have four. Instead of returning identifiers to pass back in, we return a `Service` with those values filled in for us:

code:

class WorkerApi(Service):
    def register(self, worker_id):
        return Lease(worker_id)

class Lease(Service):
    worker_id: str

    @rpc()
     def lock_queue(self, name):
        ...
        return Queue(self.worker_id, name, lock)

class Queue(Service):
    name: str
    lock: str
    worker_id: str

    @rpc()
     def get_task(self):
        return Task(.., name, lock, worker_id)

class Task(Service)
    task_id: str
    worker_id: str

    @rpc()
     def upload(self, out):
        mark_done(self.task_id, self.actions, out)

The client code looks like the desired example aboce�instead of an id string, the client gets a 'Service' response, methods included, but with some state hidden inside. The client turns this into a normal service object, and when the methods get called, that state is added back into the request. You can even add new parameters in, without changing too much of the client code.

Although the `Future` looked like a callback, returning a `Service` feels like returning an object. This is the power of self description�unlike reflection where you can specify in advance every request that can be made�each response has the opportunity to define what new requests can be made.

It's this navigation through several linked responses that distinguishes a regular command line tool from one that browses�and where REST gets its name.

The passing back and forth of requests from server to client is where the 'state-transfer' part of REST comes from, and using a common `Result` or `Cursor` object is where the 'representational' comes from. Although a RESTful system is more than just these combined. Along with a reusable browser, you have reusable proxies.

In the same way that messages describe things to the client, they describe things to the proxy too. Using GET or POST, and distinct URLs is what allows caches to work across services. using a stateless protocol (HTTP) is what allows proxying to work so effortlessly. The trick with REST is that despite HTTP being stateless, and despite HTTP being simple, you can build complex, stateful services by threading the state invisibly between smaller messages.

Although the point of REST is to build a browser, the point is to using self-description and state-transfer to allow heavy amounts of interoperation�not just a reusable client, but reusable proxies, caches, or load balancers too. Going back to the constraints, you might be able to see how they things fit together to achieve this.

Client-Server, Stateless, Caching, Uniformity, Layering and Code-on-Demand. The first, Client-Server, feels a little obvious, but sets the background. A server waits for requests from a client, and issues responses.

The second, Stateless, is a little more confusing. If a HTTP proxy had to keep track of how requests link together, it would involve a lot more memory and processing. The point of the stateless constraint is that to a proxy, each request stands alone. The point is also that any stateful interactions should be handled by linking messages together.

Caching is the third constraint, and it's back to being obvious. Requests and Responses must have some description as to if the request must be resent or if the response can be resent. The fourth constraint, Uniformity, is the most difficult, so we'll cover it last. Layering is the fith, and it means "You can proxy it".

Code-on-demand is the final, optional, and most overlooked constraint, but it covers the use of Cursors, Futures, or Parameterised Services�the idea that despite using a simple means to describe services or responses, they can be used, or run, to create new requests to send. Code-on-demand takes that further, and imagines passing back code, rather than templates and values to assemble.

With the other constraints handled, it's time for uniformity. Like statelessness, this constraint is more about HTTP than it is about the system atop, and frequently misapplied. This is the reason why people keep making database APIs and calling them RESTful, but the constraint has nothing to do with CRUD.

The constraint is broken down into four ideas: self-descriptive messages, identification of resources, manipulation of resources through representations, hypermedia as the engine of application state, and we'll take them one by one.

Self-Description is at the heart of REST, and this subconstraint fills in the gaps between the Layering, Caching, and Stateless constraints. Sort-of. It means using 'GET' and 'POST' to indicate to a proxy how to handle things, and responses indicate if they can be cached. It also means using a `content-type` header.

The next subconstraint, identification, means using different URLs for different services. In the RPC examples above, it means having a common, standard way to address a service or method, as well as one with parameters. This ties into the next constraint, which is about using standard representations across services. This doesn't mean using special formats for every API request, but using the same underlying language to describe every response. The web works because everyone uses HTML.

Uniformity so far might as well mean use HTTP (self-description), URLs (identification) and HTML (manipulation through representations), but it's the last subconstraint thats causes most of the headaches: Hypermedia as the engine of application state.

This is a fancy way of talking about how large or long requests can be broken up into interlinked messages, or how a number of smaller requests can be threaded together, passing the state from one to the next. Hypermedia referres to using `Cursor`, `Future`, or `Service` objects, application state is the details passed around as hidden arguments, and being the 'engine' means using it to tie the whole system together.

Together they form the basis of the Representational State transfer style. More than half of these constraints can be satisfied by just using HTTP�and if you digital into the thesis, you'll discover that the other half isn't about picking the right URLs, or using PUT or PATCH, but hiding those details from the end user.

REST at the end of the day is no more than a very long answer to explain what makes a web browser different, the latin name for opening a link in a new tab in a browser�the state is the part of the application you want to open (like your inbox), the repesentaton of the state is the URL in the link you're clicking on, and the transfer describes the whole back-and-forth of downloading the HTML with that URL in, and then requesting it.

Representational state transfer is what makes the back button work, why you can refresh a broken page and not have to restart from the beginning, and why you can open links in a new tab. (Unless it's twitter, where breaking the back button is but a footnote in the list of faults)

If you now find yourself understanding REST, I'm sorry. You're now cursed. Like a cross been the greek myths of Cassandra and Prometheus, you will be forced to explain the ideas over and over again to no avail. The terminology has been utterly destroyed to the point it has less meaning than 'Agile'.

Despite the well being throuroughly poisioned, these ideas of interoperability, self-description, and interlinked requests are surpisingly useful�you can break up large or slow responses, you can to browse or even parameterise services, and you can do it in a way that lets you re-use tools across services.

I haven't covered everything�there are still a few more tricks. Although a RESTful system doesn't have to offer a database like interface, it can. Along with `Service` or `Cursor`, you could imagine `Model` or `Rows` objects to return. For collection types, another trick is inlining.

Along with returning a request to make, a server can embed the result inside. A client can skip the network call and work directly on the inlined response. A server can even make this choice at runtime, opting to embed if the message is small enough.

Finally, with a RESTful system, you should be able to offer things in different encodings, depending on what the client asks for�even HTML. If you can build a reusable command line tool, generating a web interface isn't too difficult�at least this time you don't have to implement the browser from scratch.

In the end, being RESTful probably doesn't matter, your framework should take care of the details for you.

If interoperability, or common tools matter, then you might not care about the implementation details, but you should expect a little more from a RESTful system than just create, read, update and delete.

tef fucked around with this message at 09:21 on Jan 7, 2019

# ¿ Jan 7, 2019 06:49

tef: May 30, 2004; -> some l-system crap ->

prisoner of waffles posted:

he's not trying to play this role but his shticks include having a really good understanding of several ideas that enough programmers think they can explain back to him despite not understanding, ergo periodically we get queuechat or RESTchat.

I'm an old man with anger issues, also suspicious dish is a jerk, but anyway, hopefully this fills in some of the gigantic holes in my earlier rushed attempt to condense a thesis

tef fucked around with this message at 07:03 on Jan 7, 2019

# ¿ Jan 7, 2019 06:57

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›9 »