Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Happiness Commando
Feb 1, 2002
$$ joy at gunpoint $$

Chunjee posted:

Don't have a Win8 system to check with but it should be fine unless they destroyed the task tray. Out of curiosity, what's this for?

Happiness Periodic Mp3 Player

Thanks. You are awesome. I am using it as a lucid dreaming tool. I am going to run it 24/7 with a noticeable but innocuous sound of crickets chirping. During my waking hours, every time I hear the chirp, I will perform one or several reality checks to determine if I am dreaming (flying, breathing through a closed nose, etc). After several days when that behavior is programmed, I will do the same thing in my dreams. And then I will fly and I will breathe through a closed nose, and I will be able to determine that I am dreaming and gain lucidity.

Adbot
ADBOT LOVES YOU

Shaocaholica
Oct 29, 2002

Fig. 5E

SamDabbers posted:

Not true. You can use the built-in diskpart command line utility. Open a command prompt as administrator:



Thanks but I should have been more clear. You can't do it for volumes greater than 32G.

gary oldmans diary
Sep 26, 2005
I've formatted 2TB drives with FAT32Format.
Not sure why you couldn't format 16TB partitions with it at 228 64KB clusters, minus overhead.

Shaocaholica
Oct 29, 2002

Fig. 5E

Hogburto posted:

I've formatted 2TB drives with FAT32Format.
Not sure why you couldn't format 16TB partitions with it at 228 64KB clusters, minus overhead.

Its not that its not doable its just that the windows built in tools won't let you since XP apparently because they want people to move onto NTFS or some tin foil hat stuff like that. Fine by me but I need a 128G USB stick formatted fat32 because an embedded device I'm working with only reads fat32.

Chunjee
Oct 27, 2004

Happiness Commando posted:

Thanks. You are awesome. I am using it as a lucid dreaming tool. I am going to run it 24/7 with a noticeable but innocuous sound of crickets chirping. During my waking hours, every time I hear the chirp, I will perform one or several reality checks to determine if I am dreaming (flying, breathing through a closed nose, etc). After several days when that behavior is programmed, I will do the same thing in my dreams. And then I will fly and I will breathe through a closed nose, and I will be able to determine that I am dreaming and gain lucidity.

Awesome, a long time ago there was a cool lucid dreaming thread and I've been curious about it ever since. PM me if you have any success and I'll give it a shot too.

gary oldmans diary
Sep 26, 2005

Shaocaholica posted:

Its not that its not doable its just that the windows built in tools won't let you
No, I mean a lot of sites (among which there is a lot of disagreement) say the upper limit is 2TB and Wikipedia among them, but I'm wondering if that's just due to a lack of cited verification of >2TB FAT32 partitions since the math checks out up to 16TB for me.

Someone format a 4TB drive to FAT32 (You might have to specify 16+KB clusters).

Or this is getting too off-topic...

gary oldmans diary fucked around with this message at 19:40 on Mar 26, 2014

Shaocaholica
Oct 29, 2002

Fig. 5E

Hogburto posted:

Someone format a 4TB drive to FAT32 (You might have to specify 16+KB clusters).

Or this is getting too off-topic...

I just so happen to have a 4T spare I could test against but how would you verify? Just a successful format? Test with >2T of valid data?

gary oldmans diary
Sep 26, 2005

Shaocaholica posted:

I just so happen to have a 4T spare I could test against but how would you verify? Just a successful format? Test with >2T of valid data?
FAT32Format console output and Explorer reporting correct disk space should be fine.

Shaocaholica
Oct 29, 2002

Fig. 5E

Hogburto posted:

FAT32Format console output and Explorer reporting correct disk space should be fine.

If I do this you need to revise the wikipedia article.

Shaocaholica
Oct 29, 2002

Fig. 5E

Hogburto posted:

Someone format a 4TB drive to FAT32 (You might have to specify 16+KB clusters).

Ok so I went to try this and it seems that it has something to do with MBR not supporting volumes >2T. So I ended up using GPT.

...But, it didn't work:





I also tried 16K, same thing.

Shaocaholica fucked around with this message at 18:09 on Mar 27, 2014

evol262
Nov 30, 2010
#!/usr/bin/perl

Shaocaholica posted:

Ok so I went to try this and it seems that it has something to do with MBR not supporting volumes >2T.

It was using GPT, though. IIRC, Windows specifies a 2TB partition limit on FAT32 drives that Acronis ignores (and potentially gparted as well). Why you want to use an unjournaled fragmentation prone filesystem is another question entirely. Can you use UDF?

Shaocaholica
Oct 29, 2002

Fig. 5E

evol262 posted:

It was using GPT, though. IIRC, Windows specifies a 2TB partition limit on FAT32 drives that Acronis ignores (and potentially gparted as well). Why you want to use an unjournaled fragmentation prone filesystem is another question entirely. Can you use UDF?

Haha just for science. I don't want or need FAT32 for this 4T, just a test for the poster above.

gary oldmans diary
Sep 26, 2005

http://technet.microsoft.com/en-us/magazine/2006.07.windowsconfidential.aspx posted:

Note, however, that the maximum number of clusters is 0x0FFFFFFF—not the full 32-bit value of 0xFFFFFFFF. Since this is limited to a 28-bit value, programs that manipulate FAT32 file allocation tables can use the top four bits as tag bits in their internal data structures. Losing this capacity at the top end has no impact on the theoretical maximum FAT32 volume size. That’s because the FAT32 volume size is capped by the 0xFFFFFFFF maximum sector count, and at 512 bytes per sector, this results in a theoretical maximum FAT32 volume size of 2TB.
Does this drive have 4KB or 512B sectors? E: 4KB according to the product manual.
Although there are drives with 4KB sectors, I'm not sure programmers are as interested in taking them into account for the sake of overly large FAT32 drives as we are. It might just be that it's safer to hard-code a 2TB limit and save yourself some headaches between drive compatibility and OS compatibility. If Acronis let's you try it, I'm all for it, though.
If that's the end of this experiment for now, it was a good ride while it lasted.

gary oldmans diary fucked around with this message at 19:22 on Mar 27, 2014

Shaocaholica
Oct 29, 2002

Fig. 5E

Hogburto posted:

Does this drive have 4KB or 512B sectors? E: 4KB according to the product manual.
Although there are drives with 4KB sectors, I'm not sure programmers are as interested in taking them into account for the sake of overly large FAT32 drives as we are. It might just be that it's safer to hard-code a 2TB limit and save yourself some headaches between drive compatibility and OS compatibility. If Acronis let's you try it, I'm all for it, though.
If that's the end of this experiment for now, it was a good ride while it lasted.

Which Acronis tool should I be using? Free trial right?

gary oldmans diary
Sep 26, 2005

Shaocaholica posted:

Which Acronis tool should I be using? Free trial right?
Idunno. I'm just goin' on what evol262 said.

evol262
Nov 30, 2010
#!/usr/bin/perl

Shaocaholica posted:

Which Acronis tool should I be using? Free trial right?

Disk Director reputedly works. Never tried it personally.

Shaocaholica
Oct 29, 2002

Fig. 5E
Well, Acronis Disk Director won't let me format the 4T volume as fat32.

Peanut and the Gang
Aug 24, 2009

by exmarx
I want a python script to easily upload movie/tv clips to youtube. When I'm watching a hilarious episode of Psych, I want to say "Script! Make a clip of 32:03 to 32:14 of seasonXepY.avi". It'll use ffmpeg to grab that part, maybe even give me a preview, and then upload it to youtube on my account, all in the background so I can keep watching my ep and then have the youtube link ready for me when I finish the ep. Inputs would be the file name, start time, and end time. If start/end time are not in the arguments, it'll prompt for them. Please make this please.

TheDude42
Apr 23, 2003

Inappropriate text will not be synthesized.
I need a piece of software that monitors a specified Instagram user's feed, and reports back to me (let's say via email) when another specified user comments on a photo in their feed.

i.e.
User's feed to monitor: @mary
Commentor to monitor @bob
Mary posts a photo of a dog at 10:00am
Bob comments on Mary's dog photo at 2:00pm
Software sends me an email with a link to the photo, a copy of the comment, and the precise time that Bob's comment was made.

This seems difficult to monitor as Instagram doesn't seem to publish the time a comment is posted, so I assume this would require monitoring of the feed on a pretty frequent basis. But I may be woefully incorrect. The software could be cloud based, or could run on my desktop (mac) if easier.

Mantle
May 15, 2004

I was thinking IFTTT might have a trigger for instagram that would work but they don't have any triggers relating to comments. You can get a list of comments on a media via the API though: http://instagram.com/developer/endpoints/comments/

Poll that and filter for @bob.

Masked Pumpkin
May 10, 2008
I've got a Windows 2012 server with a bunch of files (400 000+) that need to be transferred to a new server. FTP has been set up, and that whole process has been smooth, except that hidden in those folders are a number of files with a space in the front: " sms.pdf", for example.

These files don't transfer happily as Windows simply doesn't like spaces in front of file names. I've looked through the available bulk rename apps, and haven't found anything that does the trick. I could do a bash script easily enough, but I'm not willing to set up cygwin on that system if I can avoid it. I'm happy to buy an avatar or forums upgrade for anyone who can do a Powershell script (or something else) to recursively search through the 200 000+ folders and remove the leading space - or recommend an app that will do it.

epswing
Nov 4, 2003

Soiled Meat

Masked Pumpkin posted:

do a Powershell script (or something else) to recursively search through the 200 000+ folders and remove the leading space - or recommend an app that will do it.

Here it is in python. Drop this into rename.py, drop rename.py into your root folder, run once and throw it away!

Python code:
import os
for folder, folders, files in os.walk(os.getcwd()):
    for file in files:
        if file.startswith(' '):
            oldpath = os.path.join(folder, file)
            newpath = os.path.join(folder, file[1:])
            os.rename(oldpath, newpath)
            print '"{0}" renamed to "{1}"'.format(file, newpath)
Edit: you have python installed right :v:

epswing fucked around with this message at 15:53 on Jun 11, 2014

ToxicFrog
Apr 26, 2008


Masked Pumpkin posted:

I've got a Windows 2012 server with a bunch of files (400 000+) that need to be transferred to a new server. FTP has been set up, and that whole process has been smooth, except that hidden in those folders are a number of files with a space in the front: " sms.pdf", for example.

These files don't transfer happily as Windows simply doesn't like spaces in front of file names.

Are you sure the problem is windows and not whatever software you're using to do the transfer? Win7, at least, is a-ok with filenames that have leading whitespace.

King Gonorrhea
Feb 11, 2008

Son of Ass Pharaoh
There may be a better thread for this but I'm hoping to just have a question answered and I think experienced app builders might know best.

My Dad wants to build/commission an image recognition app where you would take a picture, and then your phone serves up a bunch of information on what was in your picture. He has asked me to research how an app could access current databases of knowledge to assist in this, what is required etc.

I've attempted looking around but I haven't found enough information to tell me whether he's even asking the right question or not, so, at a very high level, is there any special access required for an app to consult google or wiki for information? What other storehouses of knowledge are there that may be useful in this?

Thanks for your time, a high level quick answer is fine.

Masked Pumpkin
May 10, 2008

epalm posted:

Here it is in python. Drop this into rename.py, drop rename.py into your root folder, run once and throw it away!

Python code:
import os
for folder, folders, files in os.walk(os.getcwd()):
    for file in files:
        if file.startswith(' '):
            oldpath = os.path.join(folder, file)
            newpath = os.path.join(folder, file[1:])
            os.rename(oldpath, newpath)
            print '"{0}" renamed to "{1}"'.format(file, newpath)
Edit: you have python installed right :v:

Python would be awesome, but I won't be able to install it on that system I'm afraid :(

the littlest prince
Sep 23, 2006


King Gonorrhea posted:

There may be a better thread for this but I'm hoping to just have a question answered and I think experienced app builders might know best.

My Dad wants to build/commission an image recognition app where you would take a picture, and then your phone serves up a bunch of information on what was in your picture. He has asked me to research how an app could access current databases of knowledge to assist in this, what is required etc.

I've attempted looking around but I haven't found enough information to tell me whether he's even asking the right question or not, so, at a very high level, is there any special access required for an app to consult google or wiki for information? What other storehouses of knowledge are there that may be useful in this?

Thanks for your time, a high level quick answer is fine.

Give up; image recognition is very very hard.

Alternatively, I think I recently heard about an app that likely uses amazon turk to get living breathing people to describe what is in a picture. That is the only feasible option, and someone may have beaten you to the punch (I don't have a link to it or remember the name).

I don't know whether there is an amazon turk api that you could easily call from an app, but there probably is. You would just need to mark your app as requiring internet access.

peepsalot
Apr 24, 2007

        PEEP THIS...
           BITCH!

King Gonorrhea posted:

There may be a better thread for this but I'm hoping to just have a question answered and I think experienced app builders might know best.

My Dad wants to build/commission an image recognition app where you would take a picture, and then your phone serves up a bunch of information on what was in your picture. He has asked me to research how an app could access current databases of knowledge to assist in this, what is required etc.

I've attempted looking around but I haven't found enough information to tell me whether he's even asking the right question or not, so, at a very high level, is there any special access required for an app to consult google or wiki for information? What other storehouses of knowledge are there that may be useful in this?

Thanks for your time, a high level quick answer is fine.

Sounds like google goggles

King Gonorrhea
Feb 11, 2008

Son of Ass Pharaoh
I'm not developing the app, nor do I even think it will happen, I'm aware of the difficulties of image recognition and the stiff competition. None of that was part of my question. I was asked to look in to what is required for an app to gain access to databases / information on the web at large. Any relevant help would be appreciated.

Arcsech
Aug 5, 2008

King Gonorrhea posted:

I'm not developing the app, nor do I even think it will happen, I'm aware of the difficulties of image recognition and the stiff competition. None of that was part of my question. I was asked to look in to what is required for an app to gain access to databases / information on the web at large. Any relevant help would be appreciated.

Internet access? Depends on what kind of database you're trying to talk to, who owns it, how it's configured, what format you get the data from it in... You need to know what database you're trying to access before you ask how you get information out of it.

Basically the question you're asking is way, way too vague to give any kind of useful answer, and (no offense) kind of indicates your idea of "databases" comes from TV shows. And on top of that, the hard part is the image recognition, which is such a big field of research that you should either start applying to grad school or give up. To be able to look up useful information about a thing, you'd have to get the name of the thing. And once you're able to go from "picture of thing" to "name of thing", supplying information would be pretty simple (in comparison).

Arcsech fucked around with this message at 23:02 on Jun 11, 2014

Jo
Jan 24, 2005

:allears:
Soiled Meat

King Gonorrhea posted:

There may be a better thread for this but I'm hoping to just have a question answered and I think experienced app builders might know best.

My Dad wants to build/commission an image recognition app where you would take a picture, and then your phone serves up a bunch of information on what was in your picture. He has asked me to research how an app could access current databases of knowledge to assist in this, what is required etc.

I've attempted looking around but I haven't found enough information to tell me whether he's even asking the right question or not, so, at a very high level, is there any special access required for an app to consult google or wiki for information? What other storehouses of knowledge are there that may be useful in this?

Thanks for your time, a high level quick answer is fine.

Let's break this problem into parts. I'm assuming that you're speaking of an Android app because I don't know the iPhone process very well, but there will be minimal differences in the grand scheme of things. If you mean a desktop app, that will change things. If you mean a web app, that will change things.

Your app will probably...
Take a picture.
Probably the easiest part. On Android, you specify an intent and it acquires a picture using the camera. Easy peasy.

Perform an object selection in the scene.
This requires either a user interaction to specify the items in the scene via touching/clicking, or solving the figure ground and image segmentation problem. Clicking on an image is pretty easy and not so error prone. If he _doesn't_ want that, you have to deal with figure-ground/image segmentation. Both of these are open problems in computer science and have been for probably 40 years. To give some sense of their difficulty, a team of three postdocs, a professor, and I spent a well over a year on the problem with minimal success. This will pretty much require a heuristic, but what that heuristic is depends on what the ultimate goal is. If you just want one specific object in the scene and info about it, then user input is the way to go. More recent research on multi-object classification has been done, and is currently regarded as state of the art. I'll dig up some citations if I can. This will _not_ run on a phone. At best, you can push the image to a web server and have that process the image, then return some classes for items in the scene along with bounding boxes. Hinton and LeCun are currently world leaders in this. You're not going to get any sort of universality here. Every item will be in one of perhaps 128 categories. I'll talk more about this at the end of my post.
For each object, perform either...
- An exact visual search
Which would be useful if you wanted to know _exactly_ about an object, like, for instance, the Mona Lisa
- OR perform a category lookup from the label
on Wikipedia, which would be useful if you want to know about an object with many different appearances, like a bicycle or a monkey or a sea-shell. This one would require some amount of image understanding, which is a more difficult and still unsolved problem. (See the Hinton paper at the end.)
This is the second easiest part. Wikipedia allows you to download the entire text corpus, which comes in at a cozy 15 terabytes uncompressed. I think it can be compressed down to ten gigs. You're best off just doing a search using one of many publicly available APIs. I'd recommend a Google search for [source of information] API. If that's not available, check if the source has a feed available, like an RSS feed. That will be easier to parse.

EDIT:
Here's a list of standard image classification databases used in academic research. Many of them are free to download. http://homepages.inf.ed.ac.uk/rbf/CVonline/Imagedbase.htm
Not that not all of them have ground-truth information available, and many of them are either gigantic or feature specific. There's not a lot of cross-domain success right now. If you train on one style of data, that's the style that gets used.

I've used the PASCAL and CalTech-256 database before and I think it's probably closest to what you have in mind. Take a look at the papers from the winners of yesteryear. http://pascallin.ecs.soton.ac.uk/challenges/VOC/voc2012/index.html and http://www.vision.caltech.edu/Image_Datasets/Caltech256/

This paper by Geoff Hinton is a turning point for modern computer vision. It really spurred a huge resurgence of interest in the industry. http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf Skip to the end and check out the results.

EDIT 2:
This app http://camfindapp.com/ is actually probably on the lines of what you want. I've not tried it, but it may suit what your father is looking for.

Jo fucked around with this message at 21:47 on Jun 12, 2014

Chunjee
Oct 27, 2004

Masked Pumpkin posted:

I've got a Windows 2012 server with a bunch of files (400 000+) that need to be transferred to a new server. FTP has been set up, and that whole process has been smooth, except that hidden in those folders are a number of files with a space in the front: " sms.pdf", for example.

These files don't transfer happily as Windows simply doesn't like spaces in front of file names. I've looked through the available bulk rename apps, and haven't found anything that does the trick. I could do a bash script easily enough, but I'm not willing to set up cygwin on that system if I can avoid it. I'm happy to buy an avatar or forums upgrade for anyone who can do a Powershell script (or something else) to recursively search through the 200 000+ folders and remove the leading space - or recommend an app that will do it.


I see a Powershell solution here but I also did a quick autohotkey script that works perfectly in my limited experiments.

code:
;Select parent directory to search
FileSelectFolder, The_Slected_Dir , , 3, Select Folder

	;Recursively loop all files with at least one preceding space in all sub directories
	Loop, %The_Slected_Dir%\ *.*, 0, 1
	{
	;Match any file with whitespace+ in the front of its name
	RegExMatch(A_LoopFileName, "\W+(.+)", RE_FileName)
		If (RE_FileName1 != "")
		{
		;Assign a new name to one without preceding whitespace 
		NewFileName := RE_FileName1
		;Move file to the same folder it is currently in and rename it. Overwrite if necessary
		FileMove, %A_LoopFileFullPath%, %A_LoopFileDir%\%NewFileName%, 1
		}
	
	}

ExitApp
Compiled exe: Preceding Space Replacer
WARNING: Will replace any duplicate file without a preceding space in the same folder.

Edit: oh yeah archives or an avatar would be neat.

Chunjee fucked around with this message at 19:16 on Jul 8, 2014

PPills
Oct 5, 2004
I'm not sure if this is the right place but I was using this Tampermonkey script for Ebay to highlight listings that had bids, but after they changed the format, it no longer worked. So I would really appreciate it if someone could show me, a person with no coding experience, how to fix it.

code:
// ==UserScript==
// @name           eBay - Hilight Items With Bids
// @namespace      [url]http://userscripts.org/users/126140[/url]
// @include        [url]http://*.ebay.*/*[/url]
// @grant	   none
// @updateURL	   [url]https://userscripts.org/scripts/source/66089.meta.js[/url]
// @downloadURL	   [url]https://userscripts.org/scripts/source/66089.user.js[/url]
// @version	   2.2.1
// @require 	   [url]http://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js[/url]
// @description	   Hilights items that have bids with a red border and yellow background.
// ==/UserScript==


$('document').ready(function() {
	$(".bids").each(function() {
		var text = $(this).text();
		var regExp = /[0-9]+.(b|B)ids?/;
		
		if (regExp.test(text)) {
			var match = regExp.exec(text);
			var sNumBids = match[0].split(" ",1);
			var numBids = parseInt(sNumBids, 10);
			if(numBids > 0) {
				$(this).closest('table').css("border","3px solid red");
				$(this).closest('table').css("background-color","yellow");
			}
		}
	});

});

I would really appreciate it. Thanks.

Sir_Substance
Dec 13, 2013

King Gonorrhea posted:

I'm aware of the difficulties of image recognition and the stiff competition.

Since you're asking the question on an internet forum rather then talking to leading image processing experts about it, I'm not sure you do.

Have some context.

However, to answer your question, there are a wide variety of datasets on common everyday objects out there, none that I am aware of which are designed to be used for your purpose. Some of them you have to pay for, some are free. All have different access endpoints.

There are special access requirements for google, they have pages and pages of information on what kind of developer access they give for free and what they don't. Every wiki has a different access policy, you'll have to chase them up one at a time.

There is no standard, any solution you come up with will be bespoke to a data source and predicated on your data source not changing it's format. Bad luck.

Sebbe
Feb 29, 2004

PPills posted:

I'm not sure if this is the right place but I was using this Tampermonkey script for Ebay to highlight listings that had bids, but after they changed the format, it no longer worked. So I would really appreciate it if someone could show me, a person with no coding experience, how to fix it.

code:
- snip -
I would really appreciate it. Thanks.


Here you go, here's an updated version for you: Install - View source

PPills
Oct 5, 2004

Oh my gawd. Thank you very much! Words cannot express how much I appreciate this.

Now, I assume this would not be compatible with the old layout design? Ebay had changed it to this new design a couple of months back for a very short period of time, then reverted to the old one. This would be the second time I've seen the layout changed, and I have a feeling it is here to stay. When will people learn, if it ain't broke, don't fix it? I gathered from their forum that most people don't like the new layout and even found it causing problems for their browsers.

Also, I appreciate the fact that your script doesn't prevent the item refinements box from popping up, as the old script completely broke the javascript (or whatever it's called).

P.S., looking at the source, it seems so simple, yet leaves me almost completely stumped.

Thanks again!

Sebbe
Feb 29, 2004

PPills posted:

Now, I assume this would not be compatible with the old layout design?

Hard for me to say. If you get any problems, just come and ask again. Should be easy enough to fix.

PPills posted:

Also, I appreciate the fact that your script doesn't prevent the item refinements box from popping up, as the old script completely broke the javascript (or whatever it's called).

I went and simplified the script a bit, so that may be what made it more compatible. :)

Happy to help.

Spekkio
Sep 23, 2005

Really satisfying music!
Don't know how 'tiny' of an app this would be, but maybe someone has a suggestion.

I need to take part of an excel spreadsheet that looks like this:



and turn it into this:



The green represents accommodation, the red represents vacancy.
There is usually an identifier at the start of the block that reads 'IN' or 'OUT' or a variation on that.

In the past I've written an VBA macro to read each cell left to right and build an array of dates and the in/out positions. But it was time consuming and isn't easily adopted to spreadsheets in another format.

Any ideas?

Sir_Substance
Dec 13, 2013

Spekkio posted:

The green represents accommodation, the red represents vacancy.
There is usually an identifier at the start of the block that reads 'IN' or 'OUT' or a variation on that.

In the past I've written an VBA macro to read each cell left to right and build an array of dates and the in/out positions. But it was time consuming and isn't easily adopted to spreadsheets in another format.

Any ideas?

That kind of task wouldn't be very big, I have a python script somewhere that does a similar thing to a different set of data. The key unknown is how hard it is to access excel data. I cheated and exported as a CSV, but that won't export your color data. Since you can't rely on people to put in and out at the start of the block either, it's trickier.

Is this going to be a constant thing you're running every week, or do you have a single dataset you have to convert once that's just really big?

Spekkio
Sep 23, 2005

Really satisfying music!

Sir_Substance posted:

That kind of task wouldn't be very big, I have a python script somewhere that does a similar thing to a different set of data. The key unknown is how hard it is to access excel data. I cheated and exported as a CSV, but that won't export your color data. Since you can't rely on people to put in and out at the start of the block either, it's trickier.

Is this going to be a constant thing you're running every week, or do you have a single dataset you have to convert once that's just really big?

It's not an every week job. But it's often enough that automating it would make life easier.

And every time it comes around, the data is slightly different... Sometimes there might be a 1 cell gap when the month clicks over from 31/08 to 01/09 for example. Or there may be a cell for Bob's salutation and gender.

So it would need to be something easily configurable.

Adbot
ADBOT LOVES YOU

Sir_Substance
Dec 13, 2013
That's a classic data quality problem you have there. Talk to the people who make electronic health records about trying to get consistent data out of hand written notes from mds, they know about this stuff!

If you don't feel like doing a PhD on data sanitization while you make this, you're going to have to force consistency before you start trying to process it.I would look at replacing that excel sheet with something more rigid, or at least making it a template with drop downs, and then insist that people fill it out correctly. If you don't, you'll always have to double check the results of any conversion for accuracy, in which case you've done nothing but save some wrist strain.

  • Locked thread