Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.
I'm using scala professionally at this point, so I thought we should have a thread for it and the stupid questions I constantly have.

What is scala?

quote:

Scala is an acronym for “Scalable Language”. This means that Scala grows with you. You can play with it by typing one-line expressions and observing the results. But you can also rely on it for large mission critical systems, as many companies, including Twitter, LinkedIn, or Intel do.

Scala is a type-safe functional language that runs on top of the JVM and so can access the full range of existing Java libraries. Scala calls itself object-functional: it supports common functional programming constructs (functions that take or return functions, monads, efficient recursion, immutable types, etc.) while also having a strong type system with modern OO features. Functions are themselves objects, in that:

code:
def timesTwo(x: Int): Int = x * 2
val four: Int = timesTwo(2)
Is really syntactic sugar for:

code:
object timesTwo {
  def apply(x: Int): Int = x * 2
}
val four: Int = timesTwo(2)
Who cares?

Where scala seems to really excel is in distributed environments (where immutability prevents race conditions rather than just annoying programmers) and in asynchronous environments where functional constructs like Future allow you to build non-blocking systems without needing to explicitly declare everything a callback method and just let the compiler figure it all out.

For example:

code:
def transformDatabaseRow(id: Int, f: (RawRow) => ProcessedRow): Future[ProcessedRow] = {
  val row: Future[RawRow] = getFromDatabase(id)
  row.map(f)
}
This is a function that takes as arguments an id and a function f, retrieves the row for id from the database, and applies the function to the row. However, as written transformDatabaseRow returns immediately a value that will at some point hold the processed data, the passed function isn't called yet, but will be at some point after the data becomes available. Future is a container class (a monad) for data that will exist at some point and map tells the Future to apply f to the Future's contents once they become available. Your code (f) doesn't need to care about the asynchronous nature of its application at all.

Places to learn about scala:


Other resources and major libraries:

  • Spark -- distributed HPC environment (like Hadoop but not terrible)
  • Play framework -- Reactive web framework
  • Akka Actors -- concurrent, asynchronous, and distributed messaging system
  • Slick -- Functional relationship mapping -- synchronous database access for a functional language

KernelSlanders fucked around with this message at 23:54 on Jan 10, 2015

Adbot
ADBOT LOVES YOU

PleasantDilemma
Dec 5, 2006

The Last Hope for Peace
Scala seems pretty cool. I'm getting into more Java these days and out of the php slums, Scala looks like it could be a fun next language to tackle.

KernelSlanders, what do you use Scala for at your work? How much Java did you know before you got into Scala?

As an outsider, here is my preconceptions of Scala:
  • popular for backend things
  • somewhat popular for web things
  • people complain that the language is too complex
  • I get confused every time I see val four: Int because I want to see the type come before the variable name

Pollyanna
Mar 5, 2005

Milk's on them.


I'm going through the Scala portion of Seven Languages in Seven Weeks, and is it just me, or does Swift take a bunch of concepts from Scala? :v: Realtalk though, I find it to be much more tolerable than plain Java. It's actually akin to making a project in Python or Ruby instead of "everything must be a class always always always".

PlesantDilemma posted:

Scala seems pretty cool. I'm getting into more Java these days and out of the php slums, Scala looks like it could be a fun next language to tackle.

KernelSlanders, what do you use Scala for at your work? How much Java did you know before you got into Scala?

As an outsider, here is my preconceptions of Scala:
  • popular for backend things
  • somewhat popular for web things
  • people complain that the language is too complex
  • I get confused every time I see val four: Int because I want to see the type come before the variable name

I'd love to know what Scala is used for, too. I understand that it's gotten popular for backend web dev, but where else is it applied?

Fullets
Feb 5, 2009
At my work we're using it pretty heavily for the various services surrounding our big, traditional, J2EE applications. To a lesser extent the J2EE apps are using it a bit too but the real adoption has been in backend web services. People do tend to either like it a lot more than Java or hate it utterly, but one interesting pattern that's happened is that teams that have used Scala and decide it doesn't work for them don't go back to Java; they either explore other FP-on-the-JVM things like Clojure or in a couple of cases have just switched to Haskell.

There are a couple of people I've talked to that are using it for big dataish stuff and seem to like it in that role.

It does quite remind me of C++ in regards to all the pointless minutiae that you have to learn to massage what you actually wanted to write into something that the compiler won't vomit back into your face. A lot of this isn't language complexity per se, though, so much as it is the compiler not being very good.

KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.

PlesantDilemma posted:

Scala seems pretty cool. I'm getting into more Java these days and out of the php slums, Scala looks like it could be a fun next language to tackle.

KernelSlanders, what do you use Scala for at your work? How much Java did you know before you got into Scala?

As an outsider, here is my preconceptions of Scala:
  • popular for backend things
  • somewhat popular for web things
  • people complain that the language is too complex
  • I get confused every time I see val four: Int because I want to see the type come before the variable name

As Fullets suggested, I use it for bigdataish tasks predominately: spark jobs that do product or user clustering, large scale ML, some NLP stuff, etc. It's basically used to productize data science tasks that run through cron. We also use it for internal microservice components of our web service. I've also just started using it in script mode for one-off jobs and data exploration that I used to do in python/numpy/pandas, although that's mostly to take advantage of other scala libraries we've written.

I haven't done much Java since school, although I did do some C#, which is syntactically quite similar. I fall into the like it a lot more than Java camp. I would say that it's not so much the language is too complex as that the syntax can be cryptic. I think in general it can be pretty expressive if you want it to be, although there are clearly some cases where syntax that shouldn't be needed by the compiler is. I think context bounds are a good example of that.

The type-after-variable-name syntax doesn't really bother me. I think of it as a decorator since most of the time the compiler can figure out the type implicitly and you can just leave the declaration off. Python 3 uses the same syntax for type hinting.

Cryolite
Oct 2, 2006
sodium aluminum fluoride

KernelSlanders posted:

I use it for bigdataish tasks ... spark... large scale ML... NLP... data science... data exploration that I used to do in python/numpy/pandas...

I'm learning Scala coming from C# and this is exactly the type of stuff I'd like to eventually be doing too, bonus points if it's in Scala. Is your company by any chance in the Baltimore/DC area? :allears:

I really, really want to jump ship off of .NET and maybe into a position writing Scala next, but it looks like there aren't too many companies looking for it (or at least near me).

triple sulk
Sep 17, 2014



Cryolite posted:

I really, really want to jump ship off of .NET and maybe into a position writing Scala next, but it looks like there aren't too many companies looking for it (or at least near me).

The shortest answer for this is that it isn't worth it for companies to bring on developers who don't already know Scala incredibly well. It's not a language worth using in 2015 and you'd be better served by sticking with C# or trying out one of F#, Haskell, or Clojure.

KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.

Cryolite posted:

I'm learning Scala coming from C# and this is exactly the type of stuff I'd like to eventually be doing too, bonus points if it's in Scala. Is your company by any chance in the Baltimore/DC area? :allears:

I really, really want to jump ship off of .NET and maybe into a position writing Scala next, but it looks like there aren't too many companies looking for it (or at least near me).

NYC, sorry.

I think if you want a non .NET language to break into big data projects with you should probably start with Python, to be honest. Although if you want to stay in the Baltimore/DC/Nova area you should probably look up what BAH uses. You can certainly do a lot worse than scala and I don't agree with triple sulk's suggestion of Haskel or Clojure, but he's right that very few companies are actively hiring scala developers at this point. In fact, we use "get to learn scala" as a recruiting tool since there's a perception that it may be the Next Big Thing. We've recruited a couple experienced scala devs, but one was one of the early contributors to some core language features. That said, scala will get attention on a resume, at least from us.

ultrafilter
Aug 23, 2007

It's okay if you have any questions.


Walmart Canada uses Scala for their backend, and they seem to be pretty happy with it. One of their developers gave some advice on transition to Scala that might be worth reading.

nite time dinosaur
Sep 12, 2011

this is what you get when you mess with us
There's 3 languages I use at work (primarily anyway): Java, JavaScript, and Scala. Scala is probably my favorite. The worst case scenario with Scala code, in my experience, coming from a more imperative Java background, is that I am essentially writing succinct Java programs with more immutable references, as typing "val foo" is much easier than typing "final Foo foo." The best case scenario is, a more functional style is adopted and code is easier to reason about and there is usually less chance of a weird bug and overall debugging time is less - assuming you or whoever else is proficient at reading the syntax and has some experience with and is anticipating referentially transparent (read: no side effects) methods. I will admit that the first time you see a nested map/for-comprehension with a pattern match wrapped in a Future, it's a little :wtf: but eventually you get used to it and find that a lot of problems are much easier to solve when you think of them in terms of streams.

I started learning Scala via the Coursera FP w/ Scala course, last spring. It will kick your rear end if you haven't done much FP, but you will learn a lot of useful poo poo if you manage to complete it. Some of the assignments I must've spent 11 hours to ultimately write like 5 lines of code, a humbling experience. I came to see if there was a Scala thread here as I just started reading Functional Programming in Scala and doing the exercises as I'd like to internalize some of the stuff I encountered in the Coursera class but didn't really retain.

nite time dinosaur fucked around with this message at 05:26 on Jan 21, 2015

KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.
A common programming task I run into at work is needing to process all the elements of a list into a reduced list of fewer elements such as converting web log events to sessions. We've been debating the merits of two different design patterns and wanted to see what others thought. As an example I'll use taking a string and converting it to as set of tuples that count the repeated occurrences of a character in a given run. So "foocookie!!!" becomes List((F,1), (o,2), (c,1), (o,2), (k,1), (i,1), (e,1), (s,1), (!,3)).

The first pattern is a tail recursive method that takes two arguments: first the list of already processed elements, and second the list of elements yet to be processed.

code:
def countRuns(str: String): List[(Char, Int)] = {
  def countRunsImpl(runs: List[(Char, Int)], letters: List[Char]): List[(Char, Int)] = {
    if (letters.isEmpty) runs
    else if (runs.head._1 == letters.head) countRunsImpl((runs.head._1, 
      runs.head._2 + 1) +: runs.tail, letters.tail)
    else countRunsImpl((letters.head, 1) +: runs, letters.tail)
  }

  val letters = str.toCharArray.toList
  countRunsImpl(List((letters.head, 1)), letters.tail).reverse
}
The other method is to use foldLeft with a method that has the already processed elements, and the next element to process.

code:
def countRuns(str: String): List[(Char, Int)] = {
  def countRunsImpl(runs: List[(Char, Int)], letter: Char): List[(Char, Int)] = {
    if (runs.head._1 == letter) (runs.head._1, runs.head._2 + 1) +: runs.tail
    else (letter, 1) +: runs
  }

  val letters = str.toCharArray.toList
  letters.tail.foldLeft(List((letters.head, 1)))(countRunsImpl).reverse
}
I don't think there's any performance reason to favor one over the other, it's mostly about readability. Which do you think is clearer? Is there a third pattern I'm not seeing?

e: my lines were too long or something

KernelSlanders fucked around with this message at 21:41 on Jan 31, 2015

Fullets
Feb 5, 2009
Rather than using foldLeft and reverse I'd be inclined to use foldRight as that'll be doing the same under the hood (for lists). It permits a slightly simpler version (which also has the benefit of not blowing up on empty input strings):

code:
"foocookie!!!".toList.foldRight(List.empty[(Char, Int)]) {
  case (ch, (lch, co) :: t) if ch == lch => (lch, co+1) :: t
  case (v, t) => (v,1) :: t
}

KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.
Yeah, foldRight is probably cleaner than foldLeft. I purposely didn't use an anonymous function because in reality the logic of the Implementation method would be substantially more complex.

Fullets
Feb 5, 2009
I guess the other point of note with foldRight is that if you were to switch from List to Stream or some other non strict data structure then you can run the computation lazily which on big data sets doesn't mean you have to have the whole thing in memory at the same time.

KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.
That doesn't really work though since foldRight isn't tail recursive (See: LinearSeqOptimized.scala) and using it on a stream will cause the whole stream to be read into memory. Worse yet it will do so one element per stack frame resulting in a stack overflow if the list is sufficiently large.

Fullets
Feb 5, 2009
Ah, you're quite right – I was thinking of something like the following, which seems well behaved re: the stack and is lazy but, after further thought, doesn't seem to be representable as a right fold.

code:
def countRuns[A: Equal](fa: Stream[A]): Stream[(A, Int)] = {
  def impl(fai: Stream[(A, Int)]): Stream[(A, Int)] =
    fai match {
      case (ap@(a, m)) #:: (bp@(b, n)) #:: t =>
        if (a === b) impl((a, m + n) #:: t)
        else ap #:: impl(bp #:: t)
      case t => t
    }
  impl(fa.strengthR(1))
}

leftist heap
Feb 28, 2013

Fun Shoe
Good god Scala is an ugly, ugly language.

wetfeet2000
Jan 24, 2007

It's this big
I've been following Scala for a couple of years now, just as a hobby. I wanted to point out another resource for the first post, http://aperiodic.net/phil/scala/s-99/ the Ninety-Nine Prolog Scala Problems, which can help with learning some of the more pure functional aspects of Scala.

Volguus
Mar 3, 2009

rrrrrrrrrrrt posted:

Good god Scala is an ugly, ugly language.

Yea, have fun googling when you have a problem :)

TE!
Nov 26, 2003


I LOVE THE LITTLE BOYS OF CATHOLIC!! LOLZ!

nite time dinosaur posted:

There's 3 languages I use at work (primarily anyway): Java, JavaScript, and Scala. Scala is probably my favorite. The worst case scenario with Scala code, in my experience, coming from a more imperative Java background, is that I am essentially writing succinct Java programs with more immutable references, as typing "val foo" is much easier than typing "final Foo foo." The best case scenario is, a more functional style is adopted and code is easier to reason about and there is usually less chance of a weird bug and overall debugging time is less - assuming you or whoever else is proficient at reading the syntax and has some experience with and is anticipating referentially transparent (read: no side effects) methods. I will admit that the first time you see a nested map/for-comprehension with a pattern match wrapped in a Future, it's a little :wtf: but eventually you get used to it and find that a lot of problems are much easier to solve when you think of them in terms of streams.

I started learning Scala via the Coursera FP w/ Scala course, last spring. It will kick your rear end if you haven't done much FP, but you will learn a lot of useful poo poo if you manage to complete it. Some of the assignments I must've spent 11 hours to ultimately write like 5 lines of code, a humbling experience. I came to see if there was a Scala thread here as I just started reading Functional Programming in Scala and doing the exercises as I'd like to internalize some of the stuff I encountered in the Coursera class but didn't really retain.

I was actually looking at that course (https://www.coursera.org/course/progfun) and I am definitely going to be taking that on as a side project. It was probably difficult because the professor is the creator of Scala.

Cryolite posted:

I'm learning Scala coming from C# and this is exactly the type of stuff I'd like to eventually be doing too, bonus points if it's in Scala. Is your company by any chance in the Baltimore/DC area? :allears:

I really, really want to jump ship off of .NET and maybe into a position writing Scala next, but it looks like there aren't too many companies looking for it (or at least near me).

can't pm you and you have no way to contact you in your profile.

ultrafilter
Aug 23, 2007

It's okay if you have any questions.


Odersky's intro course is really good.

Cryolite
Oct 2, 2006
sodium aluminum fluoride

TE! posted:

can't pm you and you have no way to contact you in your profile.

Fixed. You can e-mail me at ubiquitous.croak@gmail.com.

KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.

Volguus posted:

Yea, have fun googling when you have a problem :)

I was trying to see today if there's a subtle difference between ++ and ::: on List today. Google doesn't like ::: as a search query it seems.

That said, being able to write val list = item1 :: item2 :: item3 :: subList1 ++ subList2 is pretty convenient.

Sedro
Dec 31, 2008

KernelSlanders posted:

I was trying to see today if there's a subtle difference between ++ and ::: on List today. Google doesn't like ::: as a search query it seems.

That said, being able to write val list = item1 :: item2 :: item3 :: subList1 ++ subList2 is pretty convenient.
You can just look at the definition of List.++. List overrides ++ to call ::: if both sides are lists. List also overrides +: to call ::

These are equivalent
code:
a ::: b
b.:::(a)
which is important because it's faster to prepend to a list.

sink
Sep 10, 2005

gerby gerb gerb in my mouf
Edit: Thank you for making a new thread!

triple sulk posted:

The shortest answer for this is that it isn't worth it for companies to bring on developers who don't already know Scala incredibly well. It's not a language worth using in 2015 and you'd be better served by sticking with C# or trying out one of F#, Haskell, or Clojure.

I don't agree with this. I'm seeing many more companies adopt Scala or begin to dabble in it. Scala's whole strategy for enterprise adoption is that it integrates well with Java (versus Clojure) and has made compromises in language design for that accommodation. There is an easy ramp up path for large organizations with a lot of legacy Java. And a lot of large companies are realizing that if they don't want dim-witted cubicle dwellers as employees, they need to start using the new JVM hotness.

Scala is the functional enterprise language. F# or Haskell might be better functional languages. But if you want a corporate job somewhere writing more functional code, then Scala is something you absolutely should get familiar with. It's an easy gateway drug.

sink fucked around with this message at 04:01 on Feb 6, 2015

Krotera
Jun 16, 2013

I AM INTO MATHEMATICAL CALCULATIONS AND MANY METHODS USED IN THE STOCK MARKET

Pollyanna posted:

I'm going through the Scala portion of Seven Languages in Seven Weeks, and is it just me, or does Swift take a bunch of concepts from Scala?

Good question! I don't know if Swift is intentionally based on Scala in any significant way, but Scala has similar ancestry -- the goals of both could be more-or-less dismissively described as "an ML with objects that interoperates with [something that isn't ML]," which inevitably creates a lot of similarity. Languages you can look at if you want something closer to the source are Standard ML or Haskell.

Mr Shiny Pants
Nov 12, 2012

sink posted:

Edit: Thank you for making a new thread!


I don't agree with this. I'm seeing many more companies adopt Scala or begin to dabble in it. Scala's whole strategy for enterprise adoption is that it integrates well with Java (versus Clojure) and has made compromises in language design for that accommodation. There is an easy ramp up path for large organizations with a lot of legacy Java. And a lot of large companies are realizing that if they don't want dim-witted cubicle dwellers as employees, they need to start using the new JVM hotness.

Scala is the functional enterprise language. F# or Haskell might be better functional languages. But if you want a corporate job somewhere writing more functional code, then Scala is something you absolutely should get familiar with. It's an easy gateway drug.

I don't know if scala is the better enterprise language than F#. F# has the complete .Net ecosystem behind it.

comedyblissoption
Mar 15, 2006

F# seems cool but I think it still has this ridiculous limitation where the order of source files in your project must be organized to satisfy the compiler rather than your organizational needs.
http://www.sturmnet.org/blog/2008/05/20/f-compiler-considered-too-linear
You'd have to balance .Net interoperability with the coolness of other languages in the same family like Haskell.

sink
Sep 10, 2005

gerby gerb gerb in my mouf

Mr Shiny Pants posted:

I don't know if scala is the better enterprise language than F#. F# has the complete .Net ecosystem behind it.

I get the sense that corporations are making more strong bets on the JVM than the CLR. I don't have numbers to prove that, but this is my reflection from job hunting in NYC and SF. Although I do run into a a surprising number of .NET shops here and there. Stackoverflow and Zocdoc being two headliners. Do you know if they are jumping into F#?

KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.

sink posted:

I get the sense that corporations are making more strong bets on the JVM than the CLR. I don't have numbers to prove that, but this is my reflection from job hunting in NYC and SF. Although I do run into a a surprising number of .NET shops here and there. Stackoverflow and Zocdoc being two headliners. Do you know if they are jumping into F#?

My experience has been the same. I think generally startups want to avoid the Microsoft stack since it's considerably more cost up front. An m3 medium instance on AWS running the windows stack is three times the cost of one running linux. Also linux/akka-http/postgres is only one step removed form linux/django/postgres making it somewhat of an easier migration to dabble in it, for example, for micro-services where CLR requires a completely new stack.

sink
Sep 10, 2005

gerby gerb gerb in my mouf
Is akka-http usable right now (both client and server) or should I stick to Spray?

DholmbladRU
May 4, 2006
Can anyone recommend a nice scala JDBC wrapper? I saw SalikeJDBC, but the number of usages on maven is pretty low.

KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.
Slick was a long series of headaches for us and we ended up just rolling our own.

DholmbladRU
May 4, 2006

KernelSlanders posted:

Slick was a long series of headaches for us and we ended up just rolling our own.

Im not quite to the point of creating my own relational mapping yet. Very new to scala. Ill look into using Slick for now. It seems like the only production ready type safe product Ive found.

DholmbladRU fucked around with this message at 17:07 on Feb 16, 2015

KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.
Depending on the complexity of your project, here's something to get you started if you change your mind.

code:
case class Foo(name: String, value: Int)

object Foo {

  fetchFromDb(conn: Connection): List[Foo] = {
    val sql = "SELECT name, value FROM foo_table"
    val rs = conn.prepareStatement(sql).executeQuery()

    val output: List[Foo] = List[Foo]()
    while (rs.next()) output = output :+ {
      val name = rs.getString("name")
      val value = rs.getInt("value")
      Foo(name, value)
    }

    output
  }
}
Obviously there's numerous improvements that could be made: abstract the column fetchers, return a stream instead of a list, support nulls through Option outputs, etc. but if you're just trying to get some data out, this should get you started.

Sedro
Dec 31, 2008
Here's an example of slick's "lifted embedding", which basically lets you use SQL as a monad

DholmbladRU
May 4, 2006

KernelSlanders posted:

Depending on the complexity of your project, here's something to get you started if you change your mind.

code:
case class Foo(name: String, value: Int)

object Foo {

  fetchFromDb(conn: Connection): List[Foo] = {
    val sql = "SELECT name, value FROM foo_table"
    val rs = conn.prepareStatement(sql).executeQuery()

    val output: List[Foo] = List[Foo]()
    while (rs.next()) output = output :+ {
      val name = rs.getString("name")
      val value = rs.getInt("value")
      Foo(name, value)
    }

    output
  }
}
Obviously there's numerous improvements that could be made: abstract the column fetchers, return a stream instead of a list, support nulls through Option outputs, etc. but if you're just trying to get some data out, this should get you started.

That is very similar to implementation for JDBC which I would write in java. I guess the benifit of something like Slick is it can be non-blocking, as I understand it.

The documentation for the Slick Schma generator is a little spotty. I am having a hard time figuring out how to implement it.

Sedro
Dec 31, 2008
JDBC isn't async so slick can't be async (of course you can still wrap the code in Future { ... }). They're working on something for 3.0 which involves replacing JDBC.

Ganondork
Dec 26, 2012

Ganondork
Slick can be such a PITA, and isn't very friendly for Scala newcomers.

I haven't used jOOQ before, but it seems pretty awesome. The DSL is really attractive.

Anyone have experience with jOOQ?

Adbot
ADBOT LOVES YOU

Grim Up North
Dec 12, 2011

I've tried Slick and disliked it. I'm now using Squeryl in production and it's pretty nice. I'm not sure if it's still developed though. I guess for my next crazy project I'll try doobie :420:.

  • Locked thread