|
I have a CSV file with IPv4 ranges in it, that I built from IP2Location's IP2Location LITE IP-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE Database: https://lite.ip2location.com/database/db5-ip-country-region-city-latitude-longitude (Download -> IPv4 Database -> CSV Format) This is the closest I can get the file into the format that I need (5 example lines): 62.56.193.0,62.56.194.255,Pinar de Chamartin,Madrid Comunidad de,ES,40.479030,-3.668360 62.56.195.0,62.56.195.255,Lora del Rio,Andalucia,ES,37.658960,-5.527510 62.56.196.0,62.56.196.255,Lodosa,Navarra Comunidad Foral de,ES,42.422690,-2.077410 62.56.197.0,62.56.197.255,San Vicente de la Barquera,Cantabria,ES,43.385090,-4.399340 62.56.198.0,62.56.198.255,Gandia,Valenciana Comunidad,ES,38.966667,-0.183333 What I need, is to convert each line above, into 256 separate lines, so that there is a single IP address per line. (That would convert the above 5 line example into 1280 lines.) So for example, let's take one line from the above: 62.56.196.0,62.56.196.255,Lodosa,Navarra Comunidad Foral de,ES,42.422690,-2.077410 I need to turn that one line, into the following 256 lines: 62.56.196.0,Lodosa,Navarra Comunidad Foral de,ES,42.422690,-2.077410 62.56.196.1,Lodosa,Navarra Comunidad Foral de,ES,42.422690,-2.077410 62.56.196.2,Lodosa,Navarra Comunidad Foral de,ES,42.422690,-2.077410 62.56.196.3,Lodosa,Navarra Comunidad Foral de,ES,42.422690,-2.077410 ... 62.56.196.255,Lodosa,Navarra Comunidad Foral de,ES,42.422690,-2.077410 Would would be best, is some kind program or script, that can automatically write me a new file, going line by line, converting each line into 256 separate lines with one IP address per line. (The file would end up being 50 gigs or so in size by my estimate... that's perfectly OK.) Is there anyway I can do this with an application? I'm not bright enough with programming to figure out myself how to do this with Python, Javascript, etc. I would need someone else to show me how, step by step unfortunately.
|
#
?
Sep 12, 2022 00:58
|
|
|
|
| # ? May 4, 2024 00:11 |
|
|
I made a thing that'll probably do this. This is hackjob, but it'll probably work. Download here Extract to some dir. Make sure your csv is in the same directory as the executable/dll/runtime json. Requires .NET 6 (I can target .NET Framework if need be) Invoke from command line as such: code:Let me know if it falls over and I can probably fix it. Canine Blues Arooo fucked around with this message at 09:12 on Sep 12, 2022 |
|
#
?
Sep 12, 2022 09:08
|
|
|
OMG, **THANKS SO MUCH** !!! I didn't expect this. This is great!! BUT... It didn't work, mainly because it ran out of memory. It looks like the app writes all of its output to RAM first, before going to a file, so, it runs out of RAM when processing the entire file (I have 56 GB). The other problem, is that it misses one x.x.x.255 IP per line. So for the following test list: code:The following IP's are only expanded to x.x.x.254, and are missing the last x.x.x.255 line: code:There are 49 IP's in both the first and second lists, and the cities/countries/long/lat all line up in both lists, so it appears to only go to x.x.x.254 in one IP range per source line. I believe it is the last range, if there are more than one range per line. (Check the 4th one up from the bottom in the first list, it has 1.0.116.0 - 1.0.117.255, so your tool does go to 1.0.116.255, but only to 1.0.117.254.) I checked and the results of this are repeatable (meaning it gives the exact same results every time as opposed to being a random thing.) For the RAM problem, I can just do the file in sections. But I'm not sure if there's an easy "search and replace" way to take care of the x.x.x.255 issue. If you don't feel like doing any additional work for this for free, I understand! But please PM me anyways if you have a Buy Me a Coffee or anything like that! Again, I can't thank you enough! (P.S. - I'm using the Microsoft .NET Windows Runtime 6.0.8) Bula Vinaka fucked around with this message at 12:48 on Sep 12, 2022 |
|
#
?
Sep 12, 2022 12:39
|
|
|
The off by one is a trivial fix. I can also fix the memory issue here as well relatively easily - stand by
|
|
#
?
Sep 12, 2022 16:40
|
|
|
I don't think I ever asked a question like this on a forum, reddit, etc., and had someone actually write me a custom exe like this. I can't thank you enough!
|
|
#
?
Sep 12, 2022 17:37
|
|
|
Alright, I yolo'd some fixes here: Download here Should fix the off by one error. I'm not testing against the entire IPv4 list, but I'm 80% sure the memory issue is addressed. I have no idea what performance on File.Append on a large file is (or if it even matters) so uh...hopefully it's fine. Canine Blues Arooo fucked around with this message at 18:44 on Sep 12, 2022 |
|
#
?
Sep 12, 2022 18:08
|
|
|
|
| # ? May 4, 2024 00:11 |
|
|
It works!!! Thanks so much, you're awesome.
|
|
#
?
Sep 12, 2022 23:39
|
|