Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
EVGA Longoria
Dec 25, 2005

Let's go exploring!

MALE SHOEGAZE posted:

it's not perl

ruby is perl

Adbot
ADBOT LOVES YOU

EVGA Longoria
Dec 25, 2005

Let's go exploring!

EVGA Longoria posted:

ruby is perl

4 letters
consonant vowel consonant consonant
both types of jewelry rocks

open your eyes, sheeple

Jabor
Jul 16, 2010

#1 Loser at SpaceChem
actually, ruby is not practical

JewKiller 3000
Nov 28, 2006

by Lowtax

kalstrams posted:

just a heads up, pycharm is 30% off atm if you thought about getting it


i don't know what this is but i'm definitely not ever going to get it if the money goes to the django software foundation

cinci zoo sniper
Mar 15, 2013




JewKiller 3000 posted:

i don't know what this is but i'm definitely not ever going to get it if the money goes to the django software foundation
pycharm is probably the best python idea available

Will Rice
Jun 6, 2006
Will Sweep!

eschaton posted:

the canonical suggestion is lex and yacc or, to be more modern, ANTLR

forums comrade Bloody had a good time doing it in Haskell with some of the interesting tools the Haskell community provides

thanks, will start failing at a combination of these soon

oh no blimp issue
Feb 23, 2011

kalstrams posted:

pycharm is probably the best python idea available

lol at your life if you ever need this

Soricidus
Oct 21, 2010
freedom-hating statist shill
seriously. if your problem is big enough to require the facilities of an ide, it's big enough to use a grown-up language whose features work with the ide, not against it

VikingofRock
Aug 24, 2008




Soricidus posted:

seriously. if your problem is big enough to require the facilities of an ide, it's big enough to use a grown-up language whose features work with the ide, not against it

Unfortunately in Kalstram's situation this probably also involves rolling their own scientific and possibly numeric / statistical libraries, as well as explaining to their advisor and collaborators why they picked some other crazy language that no one else knows instead of just sticking with the field's standard python.

cinci zoo sniper
Mar 15, 2013




VikingofRock posted:

Unfortunately in Kalstram's situation this probably also involves rolling their own scientific and possibly numeric / statistical libraries, as well as explaining to their advisor and collaborators why they picked some other crazy language that no one else knows instead of just sticking with the field's standard python.
sort of. not so much in statistics department, but writing numerical libraries myself, or even some of the more involved sciency stuff, say various filters for signals and such, would not be nice, probably not even doable since im terrible at coding that isnt just doing math in computer. or even that :v: if i remember the time where instead of transposing an array i was doing a full rotation the other way around using rot90 from numpy, and then flipping the rotated array upside down (dont ask, i know). at this point im about 4 months away from the completion of the project. so doing 180 would not be practical. as for wrt the latter, yeah - explicitly compiled language would go against desired useability here, so would commercial stuff and/or huge systems like matlab etc et al. e: this is to the point that i dont even see options other than java then, and that'd be uhh

for next thing i do though, if its math and i get to have a say about it, i might go with python only if it's a medium-small and performance-agnostic (or there are proper workstations available, not samsung laptops with 16gb ram) project

cinci zoo sniper fucked around with this message at 19:40 on Jul 4, 2016

cinci zoo sniper
Mar 15, 2013




as to pycharm - while i like it, i generally tend to do things out of sublime text since it feels snappier on my lovely laptop, and usually dont touch pycharm unless i cant reasonably quickly debug something, or im preparing a build for release

Wheany
Mar 17, 2006

Spinyahahahahahahahahahahahaha!

Doctor Rope

Soricidus posted:

seriously. if your problem is big enough to require the facilities of an ide, it's big enough to use a grown-up language whose features work with the ide, not against it

if you code, you use an ide, even it you're using a plang. smdh that i even need to say that.


"oh, you're building a circus tent instead of an apartment building? in that case, use your teeth to tighten those bolts instead of a wrench"

Valeyard
Mar 30, 2012


Grimey Drawer
i use sublime for text manipulation only, the way it should be

like taking 50k rows of text, use a find regex that matches every 50 rows, append ") or X IN (" to all selections every 50 lines

for real though, being able to place cursors on every single line (and for smaller files, using middle mouse wheel selection) is ownage as helle

pycharm: debugging, the search anywhere tool is really useful, run/debug configuations, style and formatting, refactoring tools

hobbesmaster
Jan 28, 2008

Bloody posted:

a thin layer of wrapping above some fortran libraries and all of the gotchas of plotting in matlab is not a compelling reason

not paying a king's ransom to mathworks is though

Soricidus
Oct 21, 2010
freedom-hating statist shill

Wheany posted:

if you code, you use an ide, even it you're using a plang. smdh that i even need to say that.


"oh, you're building a circus tent instead of an apartment building? in that case, use your teeth to tighten those bolts instead of a wrench"

"oh, you're building a birdhouse instead of an apartment building? doesn't matter, i want to see blueprints signed off by a qualified engineer before you pick up that hammer"

a lightweight programmer's editor like emacs/vim for plangs, an ide for serious work, this isn't hard

Luigi Thirty
Apr 30, 2006

Emergency confection port.

turns out i was in fact wrong and i did not have perspective projection working lol

trying to hook it up based on http://ogldev.atspace.co.uk/www/tutorial12/tutorial12.html and... uh, welp



i can move forward and backward but the faces are all effed up

Soricidus
Oct 21, 2010
freedom-hating statist shill

Luigi Thirty posted:

turns out i was in fact wrong and i did not have perspective projection working lol

trying to hook it up based on http://ogldev.atspace.co.uk/www/tutorial12/tutorial12.html and... uh, welp



by the way these posts are a nice palate cleanser between courses of the usual arguments, don't stop

Luigi Thirty
Apr 30, 2006

Emergency confection port.



it looks like the points are in the right spots for perspective projection but the normals are hosed up so we're seeing the cube inside out??? those faces should be hidden! i'm not sure how to fix this yet

cinci zoo sniper
Mar 15, 2013




Luigi Thirty posted:



it looks like the points are in the right spots for perspective projection but the normals are hosed up so we're seeing the cube inside out??? those faces should be hidden! i'm not sure how to fix this yet
could it be that camera location is inside cube?

Luigi Thirty
Apr 30, 2006

Emergency confection port.

kalstrams posted:

could it be that camera location is inside cube?

you know what, i think it could be something to do with a fuckup in positioning

the camera should start at (0,0,5) pointed at the cube which is at (0,0,0) but we start inside the cube when i start the application. maybe the normal is being calculated like the camera is inside the cube even though the camera isn't

e: my view matrix was screwed up but that doesn't seem to be the problem. now my camera properly starts at (0,0,5) but it still tries to render certain faces on the inside of the cube... i think the normal calculation is still hosed.

Luigi Thirty fucked around with this message at 22:25 on Jul 4, 2016

Vanadium
Jan 8, 2005

wow they sure have a lot of different services at my new job

Valeyard
Mar 30, 2012


Grimey Drawer

Luigi Thirty posted:



it looks like the points are in the right spots for perspective projection but the normals are hosed up so we're seeing the cube inside out??? those faces should be hidden! i'm not sure how to fix this yet

ynohtna
Feb 16, 2007

backwoods compatible
Illegal Hen
yer triangle windings are the wrong direction probably

Workaday Wizard
Oct 23, 2009

by Pragmatica

Will Rice posted:

I want to learn how to write terrible parsers for programming languages. Any tips on where to start?

i had fun using f# with fparsec and fparsec-pipes. bonus points for being real easy to integrate with C# (i.e. real applications :cheeky: )

gonadic io
Feb 16, 2011

>>=

ynohtna posted:

yer triangle windings are the wrong direction probably

Sapozhnik
Jan 2, 2005

Nap Ghost
I have a question about paging results sets returned by a REST server.

Let's say I'm writing a REST service. Clients can request a large list of rows, and the server will return a few initial matches along with a link where the client can find more (the link's query parameters contain all of the state required to resume the iteration), the target of that link contains more results and another link and so on. This is all well and good, except there's a problem if any other clients come along and insert or delete rows behind the client's current position: inserting a new row will cause the next page to start with a repeat of the previous page's last row, which might not be so bad, but deleting a row will cause the client to skip one record, which kinda sucks. Statelessness in the REST server is the name of the game here so the server isn't going to keep a long-running database transaction open for the client.

One way to mitigate this problem would be to use soft deletes, then add a creation and deletion timestamp to every row. No big deal, I've got some data service wrappers around my ORM, I can add that fairly easily: every query is made with reference to a particular point in time, and the data services only return records whose liveness extent (from creation time to deletion time, where a record that has not been deleted has a logical deletion time of positive infinity) contains that point. Except that a delete transaction can race with a start-of-iteration reader transaction: in between the delete timestamp being determined and the delete actually getting committed any initial result sets returned by the server will point to a hole (because the continuation URL contains the point in time T1 at which the read transaction began, which is greater than the as-yet-uncommitted delete timestamp T0).

There don't seem to be any particularly pleasant solutions to this problem. Change strategy and make the server read ahead by one page, then return a half-open range [next_begin, next_end) range of row IDs? Seems great until next_begin gets deleted out from under you and you have no idea where you are. Also, if the client can control the sort order then bracketing the result set using two row IDs leads to some really nasty SQL probably involving window functions and poo poo. Taking an exclusive lock on the table prior to timestamping the delete and committing it seems like it would work, but that's a little extreme. Blurring the liveness extents and allowing the server to return rows that were deleted "recently" before the point-in-time specified in the query string is sloppy because then you have to choose a sensible value for "recently" but I guess that's the ugly-but-effective solution.

I feel stupid because I'm sure this is a problem that was well understood and solved before I was even born, but I can't find any meaningful answers with a bit of casual Googling. Most articles about pagination in REST APIs seem blissfully unaware that this problem even exists.

Notorious b.s.d.
Jan 25, 2003

by Reene

VikingofRock posted:

What language would you recommend for janitoring scripts which are a little more complex than reasonable for find | xargs?

python, perl, and ruby are all reasonable choices for this sort of task. they're scripting languages, fiddling with unix poo poo in a hurry is what they're meant for. just use whichever one you like the most.

just don't write any more shell scripts

it's 2016, there is enough horror in the world without a new file ending in '.sh'

Notorious b.s.d.
Jan 25, 2003

by Reene

MALE SHOEGAZE posted:

it's not perl

yeah this is the bikeshedding thing again

perl shares most of the problems that mark python and ruby. it is a bad language, but it's not any worse than the others

MononcQc
May 29, 2007

Mr Dog posted:


I feel stupid because I'm sure this is a problem that was well understood and solved before I was even born, but I can't find any meaningful answers with a bit of casual Googling. Most articles about pagination in REST APIs seem blissfully unaware that this problem even exists.

https://www.citusdata.com/blog/1872-joe-nelson/409-five-ways-paginate-postgres-basic-exotic is a decent resource to see a few of the options available in the context of postgres. A lot of what you can build in terms of paginations will have equivalents in that list, but that's not all that's possible.

The actual storage mechanism you use may very well limit what's practical pagination for you.

Luigi Thirty
Apr 30, 2006

Emergency confection port.

my normals! :(

Progressive JPEG
Feb 19, 2003

Mr Dog posted:

I have a question about paging results sets returned by a REST server.

Let's say I'm writing a REST service. Clients can request a large list of rows, and the server will return a few initial matches along with a link where the client can find more (the link's query parameters contain all of the state required to resume the iteration), the target of that link contains more results and another link and so on. This is all well and good, except there's a problem if any other clients come along and insert or delete rows behind the client's current position: inserting a new row will cause the next page to start with a repeat of the previous page's last row, which might not be so bad, but deleting a row will cause the client to skip one record, which kinda sucks. Statelessness in the REST server is the name of the game here so the server isn't going to keep a long-running database transaction open for the client.

One way to mitigate this problem would be to use soft deletes, then add a creation and deletion timestamp to every row. No big deal, I've got some data service wrappers around my ORM, I can add that fairly easily: every query is made with reference to a particular point in time, and the data services only return records whose liveness extent (from creation time to deletion time, where a record that has not been deleted has a logical deletion time of positive infinity) contains that point. Except that a delete transaction can race with a start-of-iteration reader transaction: in between the delete timestamp being determined and the delete actually getting committed any initial result sets returned by the server will point to a hole (because the continuation URL contains the point in time T1 at which the read transaction began, which is greater than the as-yet-uncommitted delete timestamp T0).

There don't seem to be any particularly pleasant solutions to this problem. Change strategy and make the server read ahead by one page, then return a half-open range [next_begin, next_end) range of row IDs? Seems great until next_begin gets deleted out from under you and you have no idea where you are. Also, if the client can control the sort order then bracketing the result set using two row IDs leads to some really nasty SQL probably involving window functions and poo poo. Taking an exclusive lock on the table prior to timestamping the delete and committing it seems like it would work, but that's a little extreme. Blurring the liveness extents and allowing the server to return rows that were deleted "recently" before the point-in-time specified in the query string is sloppy because then you have to choose a sensible value for "recently" but I guess that's the ugly-but-effective solution.

I feel stupid because I'm sure this is a problem that was well understood and solved before I was even born, but I can't find any meaningful answers with a bit of casual Googling. Most articles about pagination in REST APIs seem blissfully unaware that this problem even exists.

it feels like this is a lot of complexity for what'd effectively be a handling corner case and it'd turn into a can of worms if eg a whole span of records get modified/deleted. is it really that important to avoid showing an inconsistent view across pageloads if modifications occurred in the meantime?

ultimately the simplest way (other than ignoring the problem) would probably end up looking like what you said with timestamps where a given session is treated as viewing db state as of a specific moment in time, where all modifications with timestamps after that time are omitted from the view. if the user is really expecting their view of the data to remain totally unchanged while their session is open, i think that'd be the best way of modeling that behavior in a consistent way

but then it turns into effectively keeping a complete revision history of the data, likely with modifications being stored as diffs so that you can potentially serve different timestamped views to different sessions (and then theres the question of how long to wait before those should be reconciled/flattened)

e: and ideally this would be handled internally by your choice of db anyway if its really that common of a problem

Progressive JPEG fucked around with this message at 00:04 on Jul 5, 2016

Sapozhnik
Jan 2, 2005

Nap Ghost

MononcQc posted:

https://www.citusdata.com/blog/1872-joe-nelson/409-five-ways-paginate-postgres-basic-exotic is a decent resource to see a few of the options available in the context of postgres. A lot of what you can build in terms of paginations will have equivalents in that list, but that's not all that's possible.

The actual storage mechanism you use may very well limit what's practical pagination for you.

Interesting link, wasn't aware of the ctid thing.

The solutions presented aren't super satisfactory but it's definitely good food for thought. Thanks!

Valeyard
Mar 30, 2012


Grimey Drawer

Progressive JPEG posted:

it feels like this is a lot of complexity for what'd effectively be a handling corner case and it'd turn into a can of worms if eg a whole span of records get modified/deleted. is it really that important to avoid showing an inconsistent view across pageloads if modifications occurred in the meantime?

ultimately the simplest way (other than ignoring the problem) would probably end up looking like what you said with timestamps where a given session is treated as viewing db state as of a specific moment in time, where all modifications with timestamps after that time are omitted from the view. if the user is really expecting their view of the data to remain totally unchanged while their session is open, i think that'd be the best way of modeling that behavior in a consistent way

but then it turns into effectively keeping a complete revision history of the data, likely with modifications being stored as diffs so that you can potentially serve different timestamped views to different sessions (and then theres the question of how long to wait before those should be reconciled/flattened)

e: and ideally this would be handled internally by your choice of db if its really that common of a problem

the solution to this is called memcache, or ehcache, or some kind of caching system

just let it handle everyything fo ryou

Valeyard
Mar 30, 2012


Grimey Drawer
if you have a table of data, where page 1 is the latest data

lets say you load up the page, you see the table, you see the data on page 1 about new trailers for sale, the icon for page 2 has some paginated link to page 2 in the results that feed this table

you go away for a coffee and come back 30 minutes later

you now click page 2, but since ~a page worth of new data has arrived, the data you now see on page 2 is the data from page 1 that has been pushed back

this is fine, everytime you switch page, you are getting the most recent up to date stuff

most places just let it happen

Progressive JPEG
Feb 19, 2003

im sure radium has dealt with this right??

Valeyard
Mar 30, 2012


Grimey Drawer

Progressive JPEG posted:

im sure radium has dealt with this right??

yeah, its the opposite, where newest content is on the last page

page 1 is always page 1

Gul Banana
Nov 28, 2003

Mr Dog posted:

I have a question about paging results sets returned by a REST server.

Let's say I'm writing a REST service. Clients can request a large list of rows, and the server will return a few initial matches along with a link where the client can find more (the link's query parameters contain all of the state required to resume the iteration), the target of that link contains more results and another link and so on. This is all well and good, except there's a problem if any other clients come along and insert or delete rows behind the client's current position: inserting a new row will cause the next page to start with a repeat of the previous page's last row, which might not be so bad, but deleting a row will cause the client to skip one record, which kinda sucks. Statelessness in the REST server is the name of the game here so the server isn't going to keep a long-running database transaction open for the client.

[..] I'm sure this is a problem that was well understood and solved before I was even born, but I can't find any meaningful answers with a bit of casual Googling. Most articles about pagination in REST APIs seem blissfully unaware that this problem even exists.

it is indeed a solved question, but the answer is that stable querying isn't stateless. there's maths about it somewhere. it's a classic ACID question - you want consistent, isolated reads. so the solutions are the kind of things sql servers do. snapshots copy id list to a table), transaction ids (mark stuff not-yet-deleteable), atomicity (ditch paging..)

efficiency stability and statelessness are a pick two thing, you could retain high perf but give up consistency in the face of deletes/inserts, stream/batch All data and defer to the isolation mechanisms of the data store, or keep server side state.

Luigi Thirty
Apr 30, 2006

Emergency confection port.

i was multiplying by the wrong matrix to get transform the normals.



unfortunately now we have a new problem when we're on the same Z coordinate as a cube...

fritz
Jul 26, 2003

*leans against fencepost*
*scratches*

ynohtna posted:

yer triangle windings are the wrong direction probably
*spits*

Adbot
ADBOT LOVES YOU

tef
May 30, 2004

-> some l-system crap ->

Mr Dog posted:

I have a question about paging results sets returned by a REST server.

:worms:

quote:

This is all well and good, except there's a problem if any other clients come along and insert or delete rows behind the client's current position: inserting a new row will cause the next page to start with a repeat of the previous page's last row, which might not be so bad, but deleting a row will cause the client to skip one record, which kinda sucks.

This is what is known as "how do i implement snapshot isolation". You're doing way more than pagination.


quote:

Statelessness in the REST server is the name of the game here so the server isn't going to keep a long-running database transaction open for the client.

Statlessness/REST got nothing to do with these problems. This is only a problem if you have concurrent updates.

it's representational state *transfer* too, but no-one cares

quote:

One way to mitigate this problem would be to

implement snapshot isolation


quote:

I feel stupid because I'm sure this is a problem that was well understood and solved before I was even born, but I can't find any meaningful answers with a bit of casual Googling. Most articles about pagination in REST APIs seem blissfully unaware that this problem even exists.

they just don't care and the client handles overlap, duplication, rather than implementing snapshot isolation

also 95% of the time they're only sorting on one column, and pass in the first or last column value to get the next/previous page of results

and then it's just ORDER BY column where column > 'sort_key_passed_in' b/c even if the record is gone, well, you still got a sort key

and yeah, it's usually a timestamp

quote:

some really nasty SQL probably involving window functions

it depends what your axis is but yeah, that's probably your best bet next to not giving a gently caress

  • Locked thread