Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Volguus
Mar 3, 2009
Having played with this indexer now it just occured to me: the main job of this little piece of code is to categorize the usenet posts. In newznab they chose to do it via regex. This has the advantage that one has full control of what goes where, but the disadvantage that they'd have to be maintained. I wonder if there's a better way to go about it. I mean, it's just a categorization problem and people have been able to train computers to recognize images and shapes and so on, surely it can't be that hard to categorize a bunch of text (headers, names, etc.).
Oh well, here's another project that may never be finished, but at least it could be a bit interesting.

Adbot
ADBOT LOVES YOU

Acid Reflux
Oct 18, 2004

Volguus posted:

By default the script sleeps for only 60 seconds but I assume that can be changed somewhere.

There are caveats with different tmux versions, some only work right with some scripts. The newest versions (2.3 and higher) only work in Complete Sequential mode. If you hadn't found it already, this page has the info you'll need to help you get it running. You'll probably also want to set up the PreDB/IRC scraper integration if you haven't, as well as pull the existing historical PreDB stuff:

code:
cd /var/www/nZEDb/cli

php data/predb_import_daily_batch.php 0 local true
For the Complete Sequential scripts, the sleep timer is in user_threaded.sh around line 30, "NZEDB_SLEEP_TIME":

code:
/var/www/nZEDb/misc/update/nix/screen/user_threaded.sh
I'm not sure how it works in other configurations, that's just how mine is set up. I assume yours would/should be the same if you've got a newer version of tmux.

If you want to get rid of all the hashed files that won't resolve into anything worthwhile, go to the Admin panel, Edit Site, and change "Misc->Other Retention Hours" and "Misc->Hashed Retention Hours" to something other than zero. I have mine set for an hour, and I don't think it's deleting anything I might actually want to keep at that interval. I also might be wrong. :) It's also a good idea to set your "Minimum File Size to Make a Release" to something like 2MB, just to keep it from saving all the tiny garbage files that will accumulate.

The only thing I haven't figured out yet is how to set up the backfill in tmux so that it actually...backfills. I figured it'd be easy to set it up so it would, say, "backfill to <date>, pulling <n> headers per script cycle", but apparently I'm an idiot and can't make that happen.

I'm no expert at this by any means, but this is what's been working for me so far. I'd welcome any tips or corrections from someone who's got more experience.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Volguus posted:

Having played with this indexer now it just occured to me: the main job of this little piece of code is to categorize the usenet posts. In newznab they chose to do it via regex. This has the advantage that one has full control of what goes where, but the disadvantage that they'd have to be maintained. I wonder if there's a better way to go about it. I mean, it's just a categorization problem and people have been able to train computers to recognize images and shapes and so on, surely it can't be that hard to categorize a bunch of text (headers, names, etc.).
Oh well, here's another project that may never be finished, but at least it could be a bit interesting.

I've started work on a newznab replacement a few times doing just this with TensorFlow. It seems to work OK, but I never get around to actually doing all the work required to get from a really rough prototype to something usable.

I'm not confident it would work with less maintenance than a list of regexes anyway.

Volguus
Mar 3, 2009

Thermopyle posted:

I've started work on a newznab replacement a few times doing just this with TensorFlow. It seems to work OK, but I never get around to actually doing all the work required to get from a really rough prototype to something usable.

I'm not confident it would work with less maintenance than a list of regexes anyway.

Well, not initially no. But while the newznab approach will always require maintenance, this has the potential to not need it after a while. One approach that can be taken (by both) would be to crowdsource the maintenance via either a central server (that could end up being quite expensive) or via plain old P2P. But the machine learning approach will always be superior once it gets going.

loose-fish
Apr 1, 2005
Yeah, I also played around with something like that using sklearn - it's definitely workable albeit way slower than regexes.
But then you still need to extract stuff like "part 6/103 for file X.rar" which is also done with (sometimes the same) regexes.
I think obfuscated posts are a way bigger problem for DIY indexers because the dependence on IRC channels or other external solutions.

Incessant Excess
Aug 15, 2005

Cause of glitch:
Pretentiousness
I'm using Radarr and one thing that annoys me is that it changes the folder name for download movies. Say I download Movie.Name.2017.1080p.BluRay.x264-GROUP what radarr/sabnzbd does now is:

\Data\Movies\Movie Name (2017)\Movie.Name.2017.1080p.BluRay.x264-GROUP.mkv

when I would like it to handle like this:

\Data\Movies\Movie.Name.2017.1080p.BluRay.x264-GROUP\Movie.Name.2017.1080p.BluRay.x264-GROUP.mkv

Is this even possible or does the Radarr database require the renaming of folders?

DJ Burette
Jan 6, 2010
You can change it in the settings, you won't see the option unless you turn "show advanced settings" on though.

porktree
Mar 23, 2002

You just fucked with the wrong Mexican.
Anyone else having indexer issues? For instance cat, dog and nzb.su all are missing some of last nights TV shows.

DJ Burette
Jan 6, 2010
On that note, what are peoples go to indexers at the moment? I mainly use nzb.su and had been using usenet-crawler until they recently vanished, so I'm interested in picking up another one as a backup.

Pham Nuwen
Oct 30, 2010



I'm thinking of running my own indexer at this point, I've got a spare FreeBSD box sitting around not doing much.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

I haven't noticed any problems...all my stuff downloaded last night.

As far as the indexers I use and how good they are, this is a decent overview as of right now:

Incessant Excess
Aug 15, 2005

Cause of glitch:
Pretentiousness
Is the NZBgeek VIG worth it? I'm using nzbs.org as my indexer but sometimes stuff can be a bit slow to show up there it feels like.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Incessant Excess posted:

Is the NZBgeek VIG worth it? I'm using nzbs.org as my indexer but sometimes stuff can be a bit slow to show up there it feels like.

I dunno how to decide if its "worth it". I mean, it has stuff, and you can download it, and apparently nzbhydra likes to serve up its results when sonarr/radarr ask for it. However, if I didn't have NZBGeek, I think I'd still get everything I get...but that's because I've got plenty of other indexers.

I feel like indexers are a bit of a commodity. Get a couple or three and you're fine.

Rooted Vegetable
Jun 1, 2002

porktree posted:

Anyone else having indexer issues? For instance cat, dog and nzb.su all are missing some of last nights TV shows.

I noted a few conspicuous absences from TVDB schedules in Sonarr. Wondering if that's it? (or part of it)

8-bit Miniboss
May 24, 2005

CORPO COPS CAME FOR MY :filez:
In case anyone cares to get lifetime VIP through NZB Cat, albeit at a higher price:

quote:

I figured this time I would give you guys a heads up. This black friday will be a bit different than the past, no more $5 lifetime VIP :P

So here's what we're doing starting next monday, November 6th and running until Monday November 27th at Midnight (3 full weeks):

Lifetime VIP is coming back for $50. Once it's gone it wont come back again until next black friday at the earliest (and probably at a higher price)
All other prices are staying the same BUT once November the 27th hits, VIP prices will be doubled* so get it now while the getting is good.
*with the added work necessary to keep up with popular releases we feel this increase is more than fair. More and more indexers are becoming useless, it's a full time job to stay relevant these days and we are comitted to sticking around for the long haul.

porktree
Mar 23, 2002

You just fucked with the wrong Mexican.

Heners_UK posted:

I noted a few conspicuous absences from TVDB schedules in Sonarr. Wondering if that's it? (or part of it)

For instance, searching dog and nzb.su for "Law and Order True Crime" returns almost no results. Episode 7 aired last night. Cat has full results but is still missing e07. What's nzbs.org showing as a test?

EL BROMANCE
Jun 10, 2006

COWABUNGA DUDES!
🥷🐢😬



5 in total, 2 aired (SD/720) and 3 WEB. Looks about what I’d expect, although I don’t follow that show.

Nalin
Sep 29, 2007

Hair Elf

porktree posted:

For instance, searching dog and nzb.su for "Law and Order True Crime" returns almost no results. Episode 7 aired last night. Cat has full results but is still missing e07. What's nzbs.org showing as a test?

What are you talking about? Dog has all of them. Just search for crime.

diremonk
Jun 17, 2008

Over the last couple of weeks I've been noticing that the majority of the shows I grab in SD are only in h.265. Is this happening for anyone else or do I need to do some tweaking on my NZBHydra install?

Laserface
Dec 24, 2004

diremonk posted:

Over the last couple of weeks I've been noticing that the majority of the shows I grab in SD are only in h.265. Is this happening for anyone else or do I need to do some tweaking on my NZBHydra install?

I had this too and i think its just because that is what is being released first.

I told Sonarr and Radarr to exclude anything including h265, HEVC and h.265 in the title and have been getting the regular 1080p h.264 rips since.

HEVC is awesome for size but absolutely nothing I own plays it except my desktop. even my 2 year old apple TV cant hardware decode it unless its in a .mp4 container (which the scene opts for .mkv for some reason despite not including subtitles)

Vykk.Draygo
Jan 17, 2004

I say salesmen and women of the world unite!

EL BROMANCE posted:

5 in total, 2 aired (SD/720) and 3 WEB. Looks about what I’d expect, although I don’t follow that show.

Nalin posted:

What are you talking about? Dog has all of them. Just search for crime.

I was curious what you guys were talking about, so I checked. Dog has a 720p AMZ and a 1080P AMZ, Cat has 1080p Web and a 720p HDTV. So each site has two releases, but none of the releases are the same. What is the deal with the discrepancy between Dog and Cat? I don't think I've ever actually found what I wanted on Cat.

porktree
Mar 23, 2002

You just fucked with the wrong Mexican.

Nalin posted:

What are you talking about? Dog has all of them. Just search for crime.

Ok. So maybe I was searching for “Law and Order” when I should have been using “Law & Order”. Mea Culpa.

Nalin
Sep 29, 2007

Hair Elf

porktree posted:

Ok. So maybe I was searching for “Law and Order” when I should have been using “Law & Order”. Mea Culpa.

Dog's search is not the greatest at all. It feels like if one of the terms doesn't match, it throws out the item. When I search, I always try to choose a single word, or multiple variations.

Vykk.Draygo
Jan 17, 2004

I say salesmen and women of the world unite!

Nalin posted:

Dog's search is not the greatest at all. It feels like if one of the terms doesn't match, it throws out the item. When I search, I always try to choose a single word, or multiple variations.

That's how I do searching anywhere. Use as few words as possible while being specific enough to get what you want.

MrCodeDude
Aug 31, 2005
There does seem to be a delay in the indexers getting/processing new releases.

For certain shows I've had to go to the newsgroup's IRC channel and have the release bot send me the NZB because the episode still wasn't showing up on any indexers.

I'm also finding that Dog seems to be processing the majority of releases over Cat, Geek, OMG, and Simply.

SymmetryrtemmyS
Jul 13, 2013

I got super tired of seeing your avatar throwing those fuckin' glasses around in the astrology thread so I fixed it to a .jpg

MrCodeDude posted:

There does seem to be a delay in the indexers getting/processing new releases.

For certain shows I've had to go to the newsgroup's IRC channel and have the release bot send me the NZB because the episode still wasn't showing up on any indexers.

I'm also finding that Dog seems to be processing the majority of releases over Cat, Geek, OMG, and Simply.

You can set up an IRC listen bot with nZEDb, but that's a lot of work for very little benefit (unless you're downloading a bunch of shows every week). I just keep an eye on Sonarr's calendar and manually download things that I'm looking forward to and haven't autodownloaded yet.

KingKapalone
Dec 20, 2005
1/16 Native American + 1/2 Hungarian = Totally Badass
Any block deals for Black Friday?

Dead Goon
Dec 13, 2002

No Obvious Flaws



Is there an idiots guide to getting NZB Hydra working as an indexer with Sonarr and Radarr?

Running latest versions of all programs and am on Windows 10.

Methylethylaldehyde
Oct 23, 2004

BAKA BAKA
Indexer type is generic Newznab. Look at the config and you'll see the categories for each type of item, add those in sonarr/radarr. Enable RSS, enable search.

URL is http://localhost:5075

API path is /api

Categories is 2000,2010,2020,2030,2035,2040,2045,2050,2060,5000,5030,5040
Anime is 5070

That works for me fine for Sonarr.

Radarr categories are 2000,2010,2020,2030,2035,2040,2045,2050,2060,5000,5030,5040

Methylethylaldehyde fucked around with this message at 12:23 on Nov 19, 2017

Dead Goon
Dec 13, 2002

No Obvious Flaws



Methylethylaldehyde posted:

Indexer type is generic Newznab. Look at the config and you'll see the categories for each type of item, add those in sonarr/radarr. Enable RSS, enable search.

URL is http://localhost:5075

API path is /api

Categories is 2000,2010,2020,2030,2035,2040,2045,2050,2060,5000,5030,5040
Anime is 5070

That works for me fine for Sonarr.

Radarr categories are 2000,2010,2020,2030,2035,2040,2045,2050,2060,5000,5030,5040

Thanks, I think I was not doing the categories correctly (or at all I think).

TraderStav
May 19, 2006

It feels like I was standing my entire life and I just sat down
After being on a very old version of SABnzbd for a long time I recently updated to the most recent version. Now, whenever there are missing articles, the queue just hangs. In the past, it'd just fail at some point. It's at the point now where it'll hang until I kill the entry. I've checked the options and cannot seem to find anything that makes sense to resolve.

Any ideas?

MrCodeDude
Aug 31, 2005
Is there a way to push library updates from SABnzbd/Sonarr/Radarr to Kodi?

I have three nVidia Shields running Kodi and none of them consistently keep track of new material.

I'm running both Watchdog and XBMC Library Update, but neither appear to be working 100%.

Vykk.Draygo
Jan 17, 2004

I say salesmen and women of the world unite!

MrCodeDude posted:

Is there a way to push library updates from SABnzbd/Sonarr/Radarr to Kodi?

I have three nVidia Shields running Kodi and none of them consistently keep track of new material.

I'm running both Watchdog and XBMC Library Update, but neither appear to be working 100%.

What are you using for your library database?

wolrah
May 8, 2006
what?

MrCodeDude posted:

Is there a way to push library updates from SABnzbd/Sonarr/Radarr to Kodi?

I have three nVidia Shields running Kodi and none of them consistently keep track of new material.

I'm running both Watchdog and XBMC Library Update, but neither appear to be working 100%.



Sonarr and Radarr both have built in abilities to tell Kodi to refresh the library as soon as new content is added. They also send notifications that pop up when downloads start and finish.

If you use a central database you can just have one machine refresh, though obviously notifications will have to be set up individually regardless.

Laserface
Dec 24, 2004

Kind of annoying that the scene seems to have moved to HEVC very quickly and is now pushing out episodes for shows in h264 sometimes days later. its really loving with Sonarr (or Sonarr seems really lovely at picking stuff up if its released at my desired quality any day after the actual airing)

sickbeard was ugly and old but at least it never failed to add poo poo regardless of how long it was since air date.

8-bit Miniboss
May 24, 2005

CORPO COPS CAME FOR MY :filez:

Laserface posted:

Kind of annoying that the scene seems to have moved to HEVC very quickly and is now pushing out episodes for shows in h264 sometimes days later. its really loving with Sonarr (or Sonarr seems really lovely at picking stuff up if its released at my desired quality any day after the actual airing)

sickbeard was ugly and old but at least it never failed to add poo poo regardless of how long it was since air date.

It's because 4K :filez: are released on HEVC.

Laserface
Dec 24, 2004

8-bit Miniboss posted:

It's because 4K :filez: are released on HEVC.

I get it and I see the benefit (my 3TB library would suddenly be half the size if all was reencoded) but a lot of peoples hardware doesnt natively support it.

like an Apple TV thats 2 years old doesnt support HEVC natively. some iPads dont support it natively. then you need a workhorse of a plex server to transcode it, and while my NAS was doing OK transcoding most things on the fly it absolutely chokes on HEVC (its ARM based but still, its also not incredibly old or slow)

I mean h264 wasnt widely adopted for a long time.

EL BROMANCE
Jun 10, 2006

COWABUNGA DUDES!
🥷🐢😬



I’ve not seen any slowdown in x264 releases at all.

Nairbo
Jan 2, 2005

Laserface posted:

Kind of annoying that the scene seems to have moved to HEVC very quickly and is now pushing out episodes for shows in h264 sometimes days later. its really loving with Sonarr (or Sonarr seems really lovely at picking stuff up if its released at my desired quality any day after the actual airing)

sickbeard was ugly and old but at least it never failed to add poo poo regardless of how long it was since air date.

Are you auto DLing just 4k? Scene and p2p don't do HEVC for anything but 4k

Adbot
ADBOT LOVES YOU

Gromit
Aug 15, 2000

I am an oppressed White Male, Asian women wont serve me! Save me Campbell Newman!!!!!!!
About once a month sabnzbd will throw up an error "Unpacking failed, unable to find XXXX.partNN.rar". The completed file is sitting in a folder named "_UNPACK_filename" but is named wrong, so I just go in and rename it and I'm all good. This isn't a huge problem, I'm just wondering if anyone knows why it might happen every now and again? Win 7 and latest version 2.3.1, plenty of disk space, no SMART errors.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply