<?PHP questions that don\\'t need their own thread ?>

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > <?PHP questions that don\\'t need their own thread ?>

Acidian: Nov 24, 2006

Anyone know a good site for PHP excercises for beginners. I am trying to learn the language, but I am using a book + w3schools + a learn programming app, but none of them have any exercises. I can't learn from just reading and testing some premade examples. I am at a part where I need to start learning classes, but I feel like I have forgotten half the chapter about loops, switch and if statements.

# ¿ Mar 11, 2018 13:49

Adbot: ADBOT LOVES YOU

# ¿ May 15, 2024 10:17

Acidian: Nov 24, 2006

DarkLotus posted:

I think Laracasts is great for beginners, also Laravel is a great framework so you can't go wrong.
Jeffrey has some great videos and examples for PHP beginners too. Some are free, check it out.
https://laracasts.com/

Beyond that, I am self taught so don't have any other advice on guides or learning. I wanted to build something specific and learned how to do specific tasks along the way.
Everyone has a different learning style, I'm a hands-on learner and don't do well reading books. Watching videos at Laracasts and digging into the code from the git repo is a good way to see how things work and then adapt that to what you're trying to do.

That's kinda the way I want to go too, I just want to get my head around the basics. Problem solving is definitely the best way to learn (and hence why I want some exercises to do).

To do what I want, I need to learn both PHP and SQL. I want to download and parse (and format) informaton from xml files or flat .txt files that I download form both an xml api and an ftp, then put that information into an SQL database. Then I need to create my own interface to a website running the php/sql server, so I need to know some html (which I know), css (I know a little) and javascript (not at all). So it's a long process.

I wasn't planning on using a framework to start, as I will be doing it all internally, and the server wont be accessible for anyone outside to start with. Though to be honest I don't completely know what a framework is, other than being able to create websites without reinventing the wheel in code everytime.

I chose php over a normal programing language because I will be working in php in the future on other projects (like making modules for Magento 2.0 (which is a framework?) if I ever get that far), also in far future work I might be in a situation where I am not allowed to install and use local applicatons in python, but applications running serverside in browser are fine (doing bioinformatics research in a hospital), and anyway I am already working a bit in HTML and CSS in my current work.

# ¿ Mar 11, 2018 17:19

Acidian: Nov 24, 2006

McGlockenshire posted:

... or just use the built in REPL by invoking php -a in a terminal.

Avoid w3schools for anything related to PHP. It took a massive public shaming campaign for them to even begin to clean up just some the horrible advice that's accumulated there over the years. They are not to be trusted. Further, any general tutorial about PHP made more than five years ago should be ignored as well, as should any that don't have publishing dates at all. The PHP language and ecosystem have transformed dramatically in the past few years, especially with regard to popular frameworks.

You are right to learn the language itself, and all the built in stuff, before looking at frameworks. Something I might recommend to you is reviewing the No Framework Tutorial, which walks you through the individual architectural bits that frameworks provide and demonstrates both standalone and framework libraries you can use. Another resource once you already know the basics of the language would be PHP: The Right Way. Both of these sites assume that you're a veteran coder and that PHP is not your first language. They can be quite overwhelming to take in.

For exercises, you might consider looking for general programming exercise sites, which will at least get you familiar with the syntax. Unfortunately the name of the one I played with for a while now escapes me, but many of them that I've looked at allow you to use PHP.

e: After a re-read, are you actually completely new to programming? Without intending to insult you in any way, head on over to code.org and play with their tools for schoolchildren. It's seriously a great introduction into the process of writing code, even if all you think you're doing is connecting puzzle pieces together. Please don't learn about objects & object oriented stuff ("classes") until loops and control structures aren't awkward for you.

I took a course in java, had some python in my bioinformatics course, and I also took a course in webdesign back when html strict was a thing. However, it's been a few years so I would consider myself as pretty new to programming. I will have to design some more exercises for myself to be more familiar with the different loops, before I move on to classes. I have worked with objects before, so it will probably get back to me. I am also aware of the rapid changes in the field (it's the same with my primary field), so I stick to material that's written for PHP 5. Thanks for the heads up about w3schools, I was only using it as a second repetition of what I am reading in my book, but I think it's better to just reread the chapter I was on. I will definitely check out those links once I am more comfortable with the basics (and I will probably start learing SQL before I move on to frameworks as well).

Thank you.

rt4 posted:

Be prepared to call in paid help with Magento. It's notoriously troublesome to program. Even just hosting it can be difficult due to its performance characteristics.

No worries, we already have paid help, and they will take care of the hosting. Magento 2.0 is supposedly alot better and different than 1.0, depending on what you have heard, but I just want to know how it works since I will be working in the environment.

Murrah posted:

In case you hadn't seen it just as a side note, Repl.it is a great website/resource that allows you launch a REPL in many languages including PHP if you want to try something out quickly

I downloaded a wamp package from wampserver.com, it has Apache2, PHP and MySQL, just to spare me the effort to set it up. Then I use Dreamweaver from Adobe :filez:

to run the code without needing to publish it (I tried Eclipse and Netbeans, but didn't like it). Did not know about repl.it, but thanks, now I have a place to test code from anywhere.

# ¿ Mar 11, 2018 22:07

Acidian: Nov 24, 2006

Ranzear posted:

I strongly recommend dumping Apache entirely and forever for nginx+PHP-FPM instead.

Yes, Apache and mod_php can allegedly be a little faster... ^{_{in single core environments ... with limited memory ... and avoiding simply changing default file handler limits,}} but it takes days of tuning to beat out an out-of-the-box -FPM stack. Then you have very nice nginx static serving too and a single point of configuration instead of htaccess vomit everywhere.

I might be venting. Apache likes to completely poo poo the bed on the really braindead API stuff we do and I greatly enjoy being able to say 'no Apache' on all our future projects.

And how would I go about setting up nginx+php-fpm+mysql? The reason I went for Wamp was so I didn't have to worry about any setup time or learning how to get the 3 platforms to work together. It's just install and done.

Anyway, for a purely "sit home and learn php by myself" purpose, I am sure Apache will hold up.

Edit: I found an Ubuntu tutorial, so I can use that when I want to start setting up a production server. https://www.howtoforge.com/installing-nginx-with-php5-fpm-and-mysql-on-ubuntu-14.04-lts-lemp

Acidian fucked around with this message at 16:15 on Mar 12, 2018

# ¿ Mar 12, 2018 16:12

Acidian: Nov 24, 2006

Damnit, now I kinda want to set this up. I will reinstall Ubuntu on my server when I get home, and I might have some questions if I get stuck and unable to Google the answer.

Thanks for the help guys, you are great goons.

# ¿ Mar 13, 2018 08:21

Acidian: Nov 24, 2006

I went with a windows installation for the time being, seeing as all my harddrives are in ntfs and I want to be able to write to them without formating. I assume that I can't write to ntfs. Don't feel like booting windows to download a movie.

Downloaded PHP Non Thread Safe version, since I saw FastCGI mentioned in the description, was this correct? I changed php.ini to cgi.fix_pathinfo=0, but what does this do? I did not understand the explanation in the ini file, but I also don't understand what cgi is. Is there anything else I might want to change (I enabled some error message reporting and I also added xdebug)?

For the nginx config file, is this ok?

code:

location ~ \.php$ {
            fastcgi_pass   127.0.0.1:9123;
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
            include        fastcgi_params;
        }

Also, in the server {} part, I only added the top line, but what is the second line for [::]:?

code:

 listen 80 default_server;
 listen [::]:80 default_server;

I also downloaded and installed MariaDB, did not seem to require any additional configuration. My phpinfo() lists mysql as being present.

# ¿ Mar 14, 2018 18:46

Acidian: Nov 24, 2006

Alright, thanks guys!

I know EDI and code editors are very subjective. I am currently using Dreamweaver, but mostly because it allows me to get the output immediately in a split screen after saving the file, rather than having to refresh my browser. At my level it doesn't really matter much, but any recommendations? Sublime and PHP Storm seems the most recommended?

duz posted:

Also, Linux not being able to write to NTFS hasn't been true for a decade. Assuming you're even using them in the same machine instead of accessing it over the network via Samba.

It's definitely not a decade since last time I used linux. I can't remember if I was running Debian or Ubuntu, but I was sure I had problems writing to my NTFS drives. That doesn't mean there wasn't a hack for it, but does Ubuntu now support it out of the box?

# ¿ Mar 15, 2018 00:19

Acidian: Nov 24, 2006

Been testing PHPStorm for a bit now, and I really like it. The way the autocomplete works, how it shows me the variables I am inputting in functions and also when I got the run code/debugger working.

itskage posted:

I have a tutorial I give to new hires for setting up nginx+php on windows, with VSCode and the PHP extensions for debugging in VSCode, PHP Unit, Composer, NPM and Webpack (and now with setting up PHP CS+MD thanks to this thread). I'll have to remove some company specfic stuff from the thread, but if anyone's interested I can post it here.

I might be interested? But I don't know what Composer, NPM, Webpack and CS+MD is.

# ¿ Mar 20, 2018 21:16

Acidian: Nov 24, 2006

Masked Pumpkin posted:

I interested - hit us up 🙂

Edit: Seriously, I've been using Notepad++ for ages now - Aptana failed after not supporting PHP7, if there's a good IDE I'd like to hear about it!

As a complete newbie on the subject, I can only say that my extensive googling and other threads on this forum seems to indicate that PHPStorm (costs money/free trial) and NetBeans (free) to be the best IDE. The most recommended code editor I have seen is Sublime Text by a good margin, but I have also seen Visual Studio Code recommended by more than just Itskage, and also you are not the only one still using Notepad++. I have been using Sublime Text when I want to install something fast and light weight to code in when I am at a workstations at work, especially when I only want to work in HTML and CSS.

# ¿ Mar 21, 2018 09:36

Acidian: Nov 24, 2006

I have an actual programing question for once, just started on using php with mysql so this will be a stupid newbie question.

I am after the number of rows in a table in my database. I found a way to do it using the COUNT(*) command, but I have a feeling it could be done much simpler?

code:

$query = "SELECT COUNT(*) FROM table"; 
$process_query = mysqli_query($db, $sql); //$db is my predefined database connection
$fetch_array = mysqli_fetch_array($process_query); //I can't figure out to get the value without fetching it as an array
$table_count=$fetch_array[0];
echo $table_count;

Does mysql not store the table size anywhere? If this was a large database then I assume it has to actually count every row every time the request is passed?

Edit: Using mysqli since I haven't done any oop in php yet, and so I have no idea how classes work (will be getting there in a later course).

# ¿ Mar 30, 2018 08:59

Acidian: Nov 24, 2006

Wow, that's allot of help, thanks! I will just stick with COUNT() for getting the table size then.

Ditching mysqli now and starting with PDO is a bit demoralizing, took me long enough getting that figured out. I'll see what I do, maybe I will wait until I know more about how classes work, although I could probably figure it out without with allot of help from google.

I will have to read through the index post you linked, as I barely understand how indexing works (other than knowing I should be indexing my tables). I will be using allot of mysql moving forward, so the more I learn the better.

SQL injection I do not know what is yet, I assume it's similar to injecting javascript or html into a $_GET request, but the course I am on will cover it in the next chapter, and on how to defend against it.

# ¿ Mar 30, 2018 21:30

Acidian: Nov 24, 2006

Took me some time to figure out, but using PDO now.

I have a problem now, and I think I might be a little screwed.

I have a while loop that connects to a website, and because I don't want to kill the bandwith of the website or the server, I want a 1 second delay between each execution of the loop.

Loop:
1. Connect to website
2. Download page.
3. Parse page
4. Insert relevant data into SQL
5. Output progress to browser.
6. Repeat

The problem is the automatic timeout. The script might end up doing 1000 loops in total, and I can't have it timeout while it's working. I thought that if I put an echo statement as a progress tracker in the loop, then the loop would be constantly outputting somthing in each iteration. However, the script wants to complete before it outputs anything to the browser, so it timesout.

Is the only solution to extend the max execution time setting in php.ini? If I want the loop to break and output something between each iteration, will I need to use AJAX (which I don't know yet, I don't know any javascript yet).

# ¿ Apr 10, 2018 14:21

Acidian: Nov 24, 2006

-JS- posted:

but really your parse job should run on the server as a background process and report its status to the page instead.

How would I get it to run as a background process?

I will look into output buffering, seems like a good workaround.

Edit: How would I get it to run as a background process AND report it's status to the page*

I am looking at using the exec command and setting up a cron job, but that seemed alot of work for something that should be simple. I tried using output buffering, but it's not working. The browser doesn't output anything until the page is fully loaded (and I think the browser itself will timeout after 5 minutes, so I would have to limit the iterations of the loop to <300). Not sure if that is a quirk with nginx or Chrome.

Edit 2: Could I maybe set up a page redirect every 100 iterations of the loop, that refers to itself, and every time the page is loaded it checks the database for the first listing that has not been updated and does the next 100 iterations from there until all entries in the database are updated?

Acidian fucked around with this message at 16:54 on Apr 10, 2018

# ¿ Apr 10, 2018 15:41

Acidian: Nov 24, 2006

nielsm posted:

The really proper way would be to have a separate service/daemon running on the server, which accepts jobs posted via a message queue, or maybe just inserted into a database table. How to actually set up something like that depends wholly on the hosting environment, and most cheap webhosts won't support this scenario at all. (Cheap webhosts will also kill any long running processes spawned by you, so an exec() solution would likely break the same way.)

I am currently planning on running my server on AWS. I have an Ubuntu EC2 set up, and MariaDB RDS, but I haven't figured out how to get the two to talk to each other yet (basicly, I set up a security rule and a VPC, but I need to figure out the internal IP from the EC3 to the RDS, so I know what to put in my PDO credentials).

Anyway, I am not very experienced with linux, and I don't expect someone to sit down and write a 5 page explanation on how to set up a daemon like that and how to post the job requests via PHP. However, if you have any guides you can link me, that would be great.

When it comes to cost, AWS is free to a point. I think they will charge me if I start using the CPU overmuch, but for the limited use I have right now I don't think it will be a problem.

Edit: I think I figured it out. I might have some more specific questions later, but thanks for the help so far!

Acidian fucked around with this message at 08:46 on Apr 13, 2018

# ¿ Apr 12, 2018 07:09

Acidian: Nov 24, 2006

Question about mysql

If a php script is doing several calls on MariaDB over a longer period, and another script starts running that is also doing database calls on the same database and table, or if I have users doing stuff that calls information from database, how is all that handled? Does the database queue the requests, and will my scripts time out if they don't get an immediate response?

# ¿ Apr 21, 2018 20:29

Acidian: Nov 24, 2006

McGlockenshire posted:

Can you explain your concerns more? The reason I've responded this way is that your questions lead me to believe this is the XY problem in action.

At this point, the question was mostly academic in nature. I am not having any performance problems, but I started thinking about it in connection with my earlier question. If a user triggers a long running script from my website (read this long rear end xml file, write to my DB, check all lines with distrubutor DB, update/write to my DB). Then even the same user might start interacting with the database with other scripts while this script is running (or if I have a script running as a cron job).

Ranzear posted:

Note first that all of this is when writes are involved. If you're just doing simple reads and nothing is logically dependent on them you can just read willy-nilly.

Thank you both. I will keep this in mind when I am writing scripts interacting with my DB. If it's important to lock the row before read and write, then I will make sure to do that. Thanks for explaining how to set that up. It hadn't occurred to me that logical operations which are depending on values in the DB could be changed by other scripts while the main script is running.

Also, I usually open a DB connection at the start of any script and end it when the script ends. For my web pages I open the connection in the header and close it in the footer, which are shared between all my webpages, even the ones that might not even do anything with the DB.

Since operations on the DB are cached then there's also a lot of (most) interactions I don't need to worry about.

# ¿ Apr 21, 2018 22:52

Acidian: Nov 24, 2006

McGlockenshire posted:

Do all of that writing inside a single transaction. While the XML is processing and being loaded into the database, other readers on the table won't know unless you fiddle with the isolation levels. Isolation is the "I" in ACID, noted in that great post above. If you don't wrap the writes in a transaction, or wrap each write in its own transaction, other readers will end up seeing the data before you might be ready for it.

Batching multiple writes or doing large single writes in transactions is also better for performance.

You don't really need to expressly close your connection. When PHP is done executing, all networking connections are closed, unless you're using persistent connections. Don't use persistent connections.

Also, and I'm sure you already know this or have figured it out, but it's bad practice to mix code that generates output with code that performs important program operations. You don't need to stop what you're doing and rewrite the world or anything, just know that intermixing business logic with HTML generation tends to lead to shooting yourself in the foot, repeatedly, forever, with a minigun. Of nukes. Unfortunately doing things "correctly" requires a tremendous amount of learning and also introduces all kinds of other complexities. You can start by thinking about moving all of the HTML generation out into external files. Just using includes is fine.

If I wanted to write it all as a single transaction. Then I would just read the whole XML sheet into a huge prepare() statement?

I see the default for MariaDB/InnoDB is repeatable read. I am having some issues wrapping my head around it all, but I think I am ok as long as I am writing complete lines to the end of the database and any scripts reading from the DB will only be working on completed rows. However, I will be doing logic operations later on, where it will be important to lock the row while the script reads the row and then updates it.

I know I don't need to close the connection, but I have it set up in my footer anyway, I guess it's an unnecessary precaution.

So far I have not been writing much html or generating html. Everything from <html> to <body> is included in a header.php file which is included in an include.php file that starts all my html documents. then I have everything from </body> to </html> in a footer.php which also closes the DB connection. Mostly I have only been generating tables to check that my code is working properly and making some basic pages for inputting files that will be imported to the DB. I am saving all the HTML/CSS for last, as I want to do some videocourses on them (my html is fine, by my css is really bad), and I want to do a course on javascript and AJAX as well. I want to learn PHP and SQL as well as I can, so I am focusing on everything server sided for now, and trying to solve it there when possible.

I tried setting up my server to use https / ssl, but since I am signing them myself the browser kept warning me the site might be unsafe, super annoying. Seems I need to pay for the browsers to recognize my certificate. Is there any way around that? Itskage mentioned earlier that I should be using SSL on my website.

Acidian fucked around with this message at 21:11 on Apr 22, 2018

# ¿ Apr 22, 2018 21:08

Acidian: Nov 24, 2006

Alright, thank you guys, so much help, you are amazing!

# ¿ Apr 24, 2018 07:58

Acidian: Nov 24, 2006

I am trying to convert array keys to strings in a foreach loop, but I am getting a notice from PHP warning me about array to string conversion.

code:

foreach($word_check_array as $check_k => $fix_v){
        $check = strval($check_k);
        $fix = strval($fix_v);

Is there an easy way of doing this? I thought about using the array_keys function to store the keys in new array, then use those array elements to make calls on the $word_check_array above. However, it's alot of code for something that seems like it should be really simple (as simple as above, it works, it just gives me notices).

Edit: Nevermind, I am stupid! I was adding arrays to $word_check_array ($word_check_array = ['key' => 'value']), rather than += 'key' => 'value'. Frustrating, but at least I learned something.

Acidian fucked around with this message at 10:32 on Jun 3, 2018

# ¿ Jun 3, 2018 09:22

Acidian: Nov 24, 2006

First off, I just want to thank you all again for the amazing help I have gotten here earlier. I really feel you all go above and beyond in some of your responses, and it's a great help.

I have set up my own test environement for Magento 2, and I am trying to work with a PIM (Akeno) to populate products to Magento 2 via the Magento 2 REST API.

This will by my first official work project, and also the first time I will be actively using OOP in PHP. Even though I am doing this in my own free time, I feel the project will be better off if another developer can come in and fix my code if I die in a fire or something. I kinda took it upon myself to create an integration between the PIM I am setting up and Magento 2 which is being hosted by professionals who know what they are doing. So I trying to format well, comment well, and sort everything into classes and objects. I am using GuzzleHttp for this project, and I even set up a composer.json with dependencies. I feel like such a big boy!

So regarding using REST. Is there any common practice to rate limit REST requests in a script? I will set up the larger scripts (that is, scripts that make a lot of calls) to run at like 4 am on Sundays, but even so I worry that making too many requests on the REST API will reduce performance on the Magento server, while the script is running. Also, I might have to run scipts during the day that might not make thousands of requests, but it will still make requests until it's done. I can't really test this impact on performance on my own Magento test environment in a good way.

From what I understand of the REST API, I need to send a PUT or POST request on one object at a time. This means that if I have 500 000 new products, then I need to make 500 000 individual calls on the API. Is there any way of adding more information per request, so that I can maybe send 50 objects in one request? The tutorial or documentation on the Magento site did not seem to indicate that you could.

The GuzzleHttp documentation also shows how you can queue up several requests, and then send multiple requests concurrently, but is there any point to that? I would assume it's better to just work through them sequentially. I also don't understand the difference between a synchronous request and an asynchronous request.

I also worry about script runtime, so I am thinking that maybe I will populate a mysql table with products that are being updated, then I will set a limit to how many products will be done in one run and relaunch the script with shell_exec() if there are still products in the table that have not been updated in Magento. This table can then also server as a backup in case any products fail to update for any reason.

Acidian fucked around with this message at 16:10 on Jul 8, 2018

# ¿ Jul 8, 2018 16:06

Acidian: Nov 24, 2006

bigmandan posted:

Rate limiting your requests will depend on what the endpoint limits are. Check their documentation to make sure. You could setup some sort of worker queue (beastalkd, rabbitmq, etc...) to perform your requests. The worker would send up to the maximum requests in the allowed time period then "sleep" until the next run.

The documentation says there is no limit on the requests set by the API. Even so, I worry that it's a good idea to not overload the webserver with requests, especially if there are customers also browsing the site. However, I don't know enough about networking and how web servers handle requests to know if it's a problem or not.

I will sit down and learn beanstalkd, that seemed something that will be useful now and in later projects. Thank you!

bigmandan posted:

This will really depend on what their endpoint accepts. I've used/written endpoints that accept a json array of objects. You'll have to dig around in their docs and see.

As I understand the API documentation, the json formated file should only contain one object after the header. I haven't worked with restful APIs, and I am new to reading these types of documentations, so I wasn't sure if I had misunderstood something.

bigmandan posted:

If you have a lot of requests to make, sending multiple, concurrent, requests at a time could be more efficient than doing each sequentially. Using Guzzle's async would allow you to fire off a bunch of requests then have a handler do something with the response as they complete.

I do have a lot of requests to make, so to make it more efficient, I will give this a try. Thank you!

rt4 posted:

It'd be better if you could find a way to load them directly on the server instead of using HTTP. That way, you at least have a shot at batching the inserts in a transaction.

I have been giving this some thought, and it has some challenges. The way I am doing it currently, then I don't need any ftp/ssh access to the magento server (for a batch .csv file with all my products, for example), and I do not need to involve the hosting provider in installing anything on the server. They do have a service agreement that they might consider compromised by giving me access to the server, or by asking them to install scripts they don't know anything about.

I have a secondary issue, and that is with the ERP that my PIM will be integrated with as well. The only way I am allowed to interact with my ERP is through the RESTFUL API they have installed, which is thankfully the same as Magento (giving me a test system to develop on), but this means that even if I could set up a script on the magento server that talked directly to the database, I would still need a script that talks to the same API on our ERP platform. So I am thinking it's easier to just write one code for the same type of API but transacting between two different servers (I think the endpoint names might be different in some cases, but that's it).

Acidian fucked around with this message at 14:30 on Jul 10, 2018

# ¿ Jul 10, 2018 14:26

Acidian: Nov 24, 2006

I have an actual proper PHP question this time.

Using json_decode(), I have an array of nested arrays, and in theory I might be in a situation where I don't know how many child arrays there are.

With json_decode() I can chose to solve this as an object or as an associative array. One or the other doesn't seem to help me much in solving the problem.

So in theory, I need to go through each element, check if the element is a new array or not.

Then that array, might have more arrays, and I need to check each of those elements for new arrays.

Then I need to check those arrays for new arrays again.

In practice I know how many arrays there are, but to future proof the program, then I have to assume the amount of arrays and where they branch can change.

All the potential arrays should in theory be listed under $object->children_data or $array['children_data' => [new=>array]], if that is any help. However, it should seem possible to solve without that knowledge (could always use the is_array() function within a foreach loop as well).

I feel kinda stupid and I am baffled on how to solve this, but it seems like a common enough problem, so I hope you guys can help me.

# ¿ Jul 13, 2018 13:32

Acidian: Nov 24, 2006

I solved the problem by using the array_walk_recursive function, and retrieving the data I needed from each branch and endpoint, without needing to know how many arrays there are or how they are nested.

If I want to make a check that the array structure is correctly formed, then I will have to solve this problem, but right now I have a solution that will work for now.

Bruegels Fuckbooks posted:

In general you can solve problems of this type using recursion. I'll give a javascript solution.

The same thought process that can be used to count the array at the top level can be used on every array within, so just write a function that counts how many items are arrays within the json, and call that function on each individual array within.

I don't know any Javascript currently, but I think I understand your code and what you mean. I can easily count the number of arrays by using array_walk_recursive() and counting the number of 'id' fields.

I am thinking that I could do a function using $array[$x][$y][$z]. Then it would iterate through $z until it is done, do a $y++ until all the $y's were done, then $x++ until all the array elements were done. I would have to assume there can be more than 3 dimensions, but if I code up to 5 dimensions then that should be pretty future proof, I think.

Right now it's alot of work for very little gain, so I think I will have to come back to this issue at a later date when I want to improve my code.

rt4 posted:

So you've got some big blob of json that represents a graph? Is it a tree structure? What are you trying to get from this data structure?

It's a tree structure, and it's catalogue data. Each branch has data values, and each end point has data values, for example they include an id field and a label field.

code:

Could be something like this: 

	   id
       /        \
     id         id
    /  \       / | \  
   id  id    id id id
  / | \           /  \    
 id id id        id  id

# ¿ Jul 14, 2018 10:47

Acidian: Nov 24, 2006

I have gotten into a situation now where I plan to set up multiple cron jobs, but some of the scripts are potentially very time consuming, and my worry is that they might overlap. That is, new script might start running before the last one was complete. There are also certain script I want to be sure are run in a proper sequence, to check that certain conditions are in place, or put those conditions into place if they are not. One script I was running took 24 hours to complete, but I have some ideas on how to improve on this, and it wont take 24 hours every time its run, only the first time (populating tables over REST API which requires a response from the client for every line). In the case of the 24 h queue script, which on most days might take 0-5 minutes, if the cron job is set up to run every 15 minutes and the update suddenly takes an hour, then I cant have the same script running 4 times.

I have an idea, which I am pretty sure I got from this thread but I can't find where someone mentioned it, which is to set up a queue table in mysql. I am thinking a cron job php script will run and start the php scripts that are listed in the sql table one by one sequentially, and the scripts themselves will mark themselves as ongoing when they start. This way the another cron job script can be run to add jobs to the queue-table, or check if all the necessary scripts are already in the table/queue, and running or waiting to be run, and if the script is already in the table, then it will not add it again.

I am also thinking that I will find a loop in the scripts (all the scripts have a loop somewhere in the code) which can send an update to the SQL table every time it loops, to show that it's actually running. If my server crashes, maybe because of power outage, then the job table would be there with the queue in place, but no scripts would actually be running. So a cron job script would have to flush the "running" scripts from the queue, or maybe just empty the queue outright, and the way to figure out if a script is running or not is if it is continuously sending some kind of keep-alive information to the table. The down side of this is that the scripts would potentially take longer to run if they continuously send database updates. It might be smart if I could set up a loop counter, and say that the updates to the database queue-table only need to be sent every modulo 10, 100 or 1000 of the loop counter. Just so the queue flusher script knows that the script has sent a keep-alive the last 10-30 minutes or so.

Do you goons have any input on this? Is this a good idea, or are there better ways of doing this? I am switching over to production in a little over a week, so I don't have a whole lot of time if I need to learn something new.

# ¿ Aug 21, 2018 23:24

Acidian: Nov 24, 2006

bigmandan posted:

Most of what you have said seems pretty reasonable. If you want a fairly simple way to handle script(s) from running at the same time, you can use a locking mechanism. For example

https://symfony.com/doc/current/components/lock.html

If you need finer grain control on when scripts are run you may want to look into supervisord instead of cron.

I want to learn to use the Symfony framework, since the application I am using is written in symfony and I want to do some back end modifications later down the line. If what I say sounds reasonable, then I will just stick with that for now, and learn a better way when I start learning Symfony.

I have "supervisor" installed on the server for the application, not sure if that does the same as "supervisord", but right now I don't think I need "fine grain" control so I think it's ok.

Thank you.

# ¿ Aug 22, 2018 11:40

Acidian: Nov 24, 2006

rt4 posted:

You could also add flock to the front of your cron command to let the OS handle locking for you

That seems really simple, will check that out, thanks!

# ¿ Aug 25, 2018 11:58

Acidian: Nov 24, 2006

I am running into a weird memory issue with a script. It's a short looping script that uploads an image, so it continously loads in an image file, converts it to base 64 and saves it to a variable, uploads the image to a server and unsets the image file. Then the proccess repeats. Since this is in a while statement, and the variables are the same onces being used over and over, shouldn't the memory allocation (size and address) stay more or less the same with some fluctiuations of 1-2mb depending on the image size?

code:

    //Fetching 1 product in JSON form.
$product = $sql_client_products->get_products();

//Infinite loop insurance.
$i = 0;
    while($product && $i<10000){
        $product = json_decode($product['ProductJSON'], true);

        //Get file location and image file name.
        $file_loc = $product['media_gallery_entries'][0]['content']['base64_encoded_data'];
        $filen_name = $product['media_gallery_entries'][0]['content']['name'];

        //Fetch image from server.
        if($ak_client->get_image_from_url($filen_name, $file_loc)){
            $image_base64_encoded_data = base64_encode(file_get_contents($filen_name));

            //Remove the image location and instead insert the base64 image date
            $product['media_gallery_entries'][0]['content']['base64_encoded_data'] = $image_base64_encoded_data;
            unlink($filen_name);
        }
        //Uploading the product with image to database.
        if($mag_client->add_product($product)){
            //Deleting the product from database.
            $sql_client_products->del_product($product['sku']);
        }

        $date = new DateTime();
        echo $date->format('H:i:s') . ": " . ++$i . "sku: ". $product['sku'] . "\n";

	//Get new product, if table is empty, returns false.
        $product = $sql_client_products->get_products();
    }

# ¿ Sep 5, 2018 15:59

Acidian: Nov 24, 2006

bigmandan posted:

I don't see anything erroneous that would cause memory issues. Without knowing what $ak_client, $mag_client, etc. are doing under the hood it's difficult to tell.

You may want to do some profiling with xdebug and cachegrind. Depending on the version I think profiling was removed then added back into xdebug at some point though.

Ok, thanks, I will try running xdebug on it later tonight. I have also increased the max script size from 128MB to 2GB, so will see how far that gets me.

ak_client and mag_client are guzzle functions just sending a request and recieving data. Again it's the same variables being called over and over, can't see any situation where any new variables would be "piling up". I also uploaded 43000 products without images using the same class and functions.

# ¿ Sep 5, 2018 16:36

Acidian: Nov 24, 2006

Trying to enable ssl on my webserver and I am losing my mind. Sorry to pester the PHP thread about this, but there is no specific webserver thread I think.

I am trying to use letsencrypt with certbot. It refuses to write the /.well-known/acme-challenge/ folder. I have tried disabling all sites, and just made a new config file that is super basic:

Cloudfare is blocking me trying to add the following text, so I will have to add it as a photo.

Both fail. I have been at this for 3 hours now trying different configurations. The example.com domain has ssl encryption with another provider, I do not know who, but I am just trying to add some ssl encryption to the subdomain at check.example.com

In all cases, it refuses to write the /.well-known/acme-challenge/ folder. No matter where I point the configuration to or how I chmod or chown the folders.

# ¿ Sep 11, 2018 16:48

Acidian: Nov 24, 2006

Peggle Fever posted:

PHP does not do a great job with memory in a loop like this.

Try to minimize your calls to the database. Process the images, then make a single operation to the database with all the successful images. Log any errors from the images.

My first script did not use a database, but because uploading to the server takes alot of time, 4-7seconds per product depending on image size for request and response, I wanted to run one script that adds all the products to a database, which goes super fast. Then I have 5-10 scripts running separately using shell_exec() to upload the data from the database. This cut the upload time from 48 hours to under 12 hours for the whole database. Usually I will not be uploading the whole database, only the changes being made, but sometimes there might be many thousand changes and I want the changes uploaded before people start work in the morning.

Edit: If it's all about calls to the databse, then maybe I could add 10 products (need to see product description lenghts and size of the text sql field to calculate how many products I could safely add) to one row, then make 1 call, process 10 products and upload each, then make a new database call.

Acidian fucked around with this message at 16:56 on Sep 11, 2018

# ¿ Sep 11, 2018 16:53

Acidian: Nov 24, 2006

I want to zip a folder full of images. Opening an archive, adding 1 image, closing the archive works. However, it is slow, it makes a new zip file for every image, and sometimes the close() function fails and the temp file is left in the folder and the image is not added ( I assume).

Trying to open the archive before the foreach loop, does not work. It doesnt even try to make the file it seems, but the open() statement still returns TRUE. I really don't understand what is going on.

This works:

code:


$files = scandir($dir);
$image_archive = 'images.zip';

            foreach ($files as $file){
                if($file == '..' || $file == '.'){
                    continue;
                }

                $zip = new ZipArchive();
                $zip->open($image_archive, ZipArchive::CREATE);

                $file_loc = $dir . $file;
                $zip->addFile($file_loc, $file);
                $zip->close();
}

This does not work:

code:


$files = scandir($dir);
$image_archive = 'images.zip';

$zip = new ZipArchive();
$zip->open($image_archive, ZipArchive::CREATE);

            foreach ($files as $file){
                if($file == '..' || $file == '.'){
                    continue;
                }

                $file_loc = $dir . $file;
                $zip->addFile($file_loc, $file);
                }
$zip->close();

What am I doing wrong?

# ¿ Jan 20, 2019 23:41

Adbot: ADBOT LOVES YOU

# ¿ May 15, 2024 10:17

Acidian: Nov 24, 2006

Ok, thank you for checking. I will just try and do some further testing and see, and check the php.ini file for any memory restrictions.

To me it doesn't even seem to make the file, and when I tried making the file beforehand and just appending the images to the file, that did not work either.

# ¿ Jan 22, 2019 10:27

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > <?PHP questions that don\\'t need their own thread ?>