I think it would be super useful to give LNT a RESTful interface. As LNT grows, it would be nice to be able to update some of the pages with Ajax(or json). Especially for pages with long load times, it makes sense to be able to speed up load times, by fetching the data after page load. It would also allow us to build more dynamic pages, and allow non-web clients better structured access to LNT information.
We have been doing this to a degree already by adding &json=True parameters to some of the pages. This approach is ad-hoc, serialization and exploitability have been a big pain. I think it is time to formalize the non-web interface, and use some technology to make it as rich and useful as possible.
Since we are not in the business of reinventing web technologies, I think we should use one of the preexisting flask frameworks for this. I have experimented with two: Eve + eve-sqlalchmey and flask-RESTful.
If anyone in the community has some ideas on this I’d love to hear them. Thus far, I hacked on Eve for an afternoon and could not get it to understand LNTs (admittedly obscure) database structure. Flask-RESTful looked simple, but more work to build the actual interfaces, though it seems like it might be a more natural fit for LNTs URL schema.
If no one has any options, I will try and integrate flaskRESTful into LNT.
This sounds very interesting .
I’m all for not reinventing the wheel. I don’t have experience with frameworks to implement REST interfaces so can’t comment on which one is best suited for LNT.
Just a few questions off the top of my head:
Do you already have ideas for what the REST API roughly will look like?
Very close to the database structure, or something slightly higher-level?
Should the current existing URL scheme be extended to a full REST API; or do you think it’s best for the REST API to live in a separate “namespace”?
As to what technology we use, I'm not particularly fussy, but REST can
be a double edge sword, especially if your server is already fully
loaded, like LNT.
I'm not against it, by all means, I do believe it's the simplest and
most efficient of web interfaces, particularly for dynamic websites
like you propose. But we need to solve the stability problem first, or
REST+Ajax will only make it worse.
It may get the bare page quicker, but it will also load the server for
longer, and multiply the number of database connections to the server,
which is our main problem.
Once we solve the slow queries problem, and increase the number of
concurrent connections to a limit that we know we can handle, I'm all
up for using whatever RESTful technology you like.
That is a good question.
I think the API should live outside the current url namesapce, but have a similar schema (maybe /api/database/resource). Where it makes sense, lets match the url scheme for the web interface.
I think for non-web clients, they probably want basically JSONed versions of pretty much what is in the interface now (at least until they know better). The use cases I have seen are writing scripts to pull data for custom reporting dashboards, or pulling data into iPython Notebook.
I think for Ajaxing the front end, we are going to want more DB like resources, probably with query parameters.
My current motivation is getting access to the FieldChange database entries, to dynamically render them in a new interface. I would want FieldChanges + URI to anything they link to, I would also like to be able to limit/filter the queries and paginate the results. I also want to be able to construct a graph interface where data sets can be dynamically added and removed, that would be collecting sample sets, + exploring run machine data. For this in envisioned /api/database/FieldChanges with url or post parameters to set limits and filters.
I don’t want to block progress on this because of a bug on the llvm.org LNT server. That just needs to be fixed.
I don’t think the issues of scale are a problem with LNT inheriently. We run a private LNT server instance on smaller hardware that is running 5X the amount of work the llvm.org server is. It has very good uptime and no request failures, in fact I would say it could still handle 10x the load it is running at right now, maybe even 20x. In the past we have had issues with bad workloads breaking the server, and if you sit down and have access to the logs and the running server it take almost no time to sort the issue out. The llvm.org LNT server has a bug with long running requests, and that needs to be fixed.
I agree that we should be careful with the queries and number of requests. But I think in the long run this would provide a net win. We can start to build interfaces that don’ t need to pull as much data out of the database. Lots of short fast requests. In LNT we have consistently made the tradeoff of presenting all the data, since breaking up requests requires far more complicated interface design. With these apis, we can start to undo that mess!
I absolutely agree with you. I was just making the point that we need to get that solved first, at least in the public instance.
I’m away from web backend for long enough that I can’t easily solve that problem, I’m afraid.
I did try.