Spark Scala - How to create new column for each group and then explode the column values in spark dataframe -


my dataframe below col names index, words

index | words 1 | {col1: a, col2: b} 1 | {col1: c, col2: d} 2 | {col1: a, col2: b} 2 | {col1: e, col2: f}

i looking output below output df column names one,two

one | 2  {col1: a, col2: b} | {col1: a, col2: b}  {col1: c, col2: d} | {col1: e, col2: f}

any 1 please me out on one. using spark 1.5.2 version

if sure every index value there 2 words can use below :

sqlcontext.sql("select index, collect_list(words)[0],collect_list(words)[1] yourtable group index").show

Comments

Post a Comment

Popular posts from this blog

javascript - Knockout pushing observable and computed data to an observable array -

i trying push multiple data (firstname , lastname), along when compute firstname , lastname , try push data in same array thrown error. below code. viewmodel: var viewmodel = function () { var self = this; self.gameofthrones = ko.observablearray ([ {firstname: 'jon', lastname: 'snow'}, {firstname: 'robb', lastname: 'stark'} ]); self.firstname = ko.observable(''); self.lastname = ko.observable(''); (var i=0; i<self.gameofthrones().length; i++) { self.gameofthrones()[i].fullname = ko.computed (function () { return self.gameofthrones()[i].firstname+" "+self.gameofthrones()[i].lastname; }) }; $('#classic').click( function() { self.gameofthrones.push( {firstname: $('#fn').val(), lastname: $('#pr').val()} ); }); } ko.applybindings(new viewmodel()); html view: <form class="" action="index.html" method="post"...
Read more

Trouble making a JSON string -

i'm trying upgrade existing json structure more complex one. the original idea bunch of x animals , each 1 having bunch of caracteristics: animals animal 1 claws:4 eyes:2 animal 2 claws:0 eyes:6 etc. the json this: { "animals":[ {"claws":"4", "eyes":"2"}, {"claws":"0", "eyes":"6"}, etc. ] } so can see, don't name each animal object, caracteristics of animal element of array. can use them in loop animals[x].claws now want add nest each animal, like: animals animal 1 head eyes:2 ears:2 body claws:4 tails:1 legs:4 animal 2 head eyes:6 ears:0 body claws:0 tails:0 legs:8 but didn't manage without naming each animal object (with same name "animal") , using arrays what's in animal: { "animals":[ {"animal":[ {"head": ...
Read more

sitecore - Resolve ISitecoreService using SimpleInjector -

isitecoreservice accepts database name string parameter in constructor (web or master) isitecoreservice service = new sitecoreservice("master"); //or isitecoreservice service = new sitecoreservice("web"); is possible dynamically send database name parameter ioc , resolve it? example send web/master string parameter , new instance of isitecoreservice like this? container.register<isitecoreservice>(() => new sitecoreservice("master"));
Read more