Submit Application To Hadoop YARN Using C#

Hadoop 2.5.0 introduces new REST API for YARN Resource Manager to submit applications. Now we can utilize that to submit applications to YARN using non-Java programming languages, e.g. C#.

Following is an example. In my future post, I may write how to customize YARN Container Executor to execute your own logic to create application, for example, creating in-process application which runs inside Resource Manager process rather than a new process. Let's put that aside for now, and see how to submit application in C#.

Define JSON Classes for YARN REST API

YARN REST API supports JSON or XML request. Here we create several classes which are used to serialize JSON request. We use Newtonsoft JSON library.

    #region Classes for JSON serialization

    public class SubmitApplicationRequest

    {

        [JsonProperty("application-id")]

        public string ApplicationId { get; set; }

        [JsonProperty("application-name")]

        public string ApplicationName { get; set; }

        [JsonProperty("am-container-spec")]

        public AMContainerSpec AMContainerSpec { get; set; }

        [JsonProperty("unmanaged-AM")]

        public string UnmanagedAM { get; set; }

        [JsonProperty("max-app-attempts")]

        public string MaxAppAttempts { get; set; }

        [JsonProperty("resource")]

        public Resource Resource { get; set; }

        [JsonProperty("application-type")]

        public string ApplicationType { get; set; }

        [JsonProperty("keep-containers-across-application-attempts")]

        public string KeepContainersAcrossApplicationAttempts { get; set; }

    }

    public class AMContainerSpec

    {

        [JsonProperty("local-resources")]

        public LocalResources LocalResources { get; set; }

        [JsonProperty("commands")]

        public Commands Commands { get; set; }

    }

    public class LocalResources

    {

        [JsonProperty("entry")]

        public LocalResourceEntry[] Entry { get; set; }

    }

    public class LocalResourceEntry

    {

        [JsonProperty("key")]

        public string Key { get; set; }

        [JsonProperty("value")]

        public LocalResourceEntryValue Value { get; set; }

    }

    public class LocalResourceEntryValue

    {

        [JsonProperty("resource")]

        public string Resource { get; set; }

        [JsonProperty("type")]

        public string Type { get; set; }

        [JsonProperty("visibility")]

        public string Visibility { get; set; }

        [JsonProperty("size")]

        public string Size { get; set; }

        [JsonProperty("timestamp")]

        public string Timestamp { get; set; }

    }

    public class Commands

    {

        [JsonProperty("command")]

        public string Command { get; set; }

    }

    public class Resource

    {

        [JsonProperty("memory")]

        public string Memory { get; set; }

        [JsonProperty("vCores")]

        public string VCores { get; set; }

    }

    #endregion

Codes to Submit Application

The default REST service url is like http://localhost:8088/ws/v1/cluster/... It accepts POST or PUT on different APIs. We can use WebClient class in C# to either post data or put data to the server side.

Below are the sample code. It submit an application to launch notepad.exe.

class Program

    {

        public const string NewApplicationUrl = "http://localhost:8088/ws/v1/cluster/apps/new-application";

        public const string SubmitApplicationUrl = "http://localhost:8088/ws/v1/cluster/apps";

        public const string ApplicationStateUrl = "http://localhost:8088/ws/v1/cluster/apps/{0}/state";

        static void Main(string[] args)

        {

            var submitApplicationRequest = new SubmitApplicationRequest()

            {

                ApplicationId = "", // will assign real application id later

                ApplicationName = "TestApp",

                AMContainerSpec = new AMContainerSpec()

                {

                    Commands = new Commands()

                    {

                        Command = "notepad.exe"

                    }

                },

                UnmanagedAM = "false",

                MaxAppAttempts = "2",

                Resource = new Resource()

                {

                    Memory = "1024",

                    VCores = "1"

                },

                ApplicationType = "YARN",

                KeepContainersAcrossApplicationAttempts = "false"

            };

            using (WebClient client = new WebClient())

            {

                client.Headers[HttpRequestHeader.Accept] = "application/json";

                client.Headers[HttpRequestHeader.ContentType] = "application/json";

                byte[] responseBytes = client.UploadData(NewApplicationUrl, "POST", Encoding.UTF8.GetBytes(""));

                string responseString = Encoding.UTF8.GetString(responseBytes);

                Console.WriteLine(responseString);

                var json = (JObject)JsonConvert.DeserializeObject(responseString);

                var applicationId = (string)json["application-id"];

                submitApplicationRequest.ApplicationId = applicationId;

                string requestString = JsonConvert.SerializeObject(submitApplicationRequest);

                client.Headers[HttpRequestHeader.Accept] = "application/json";

                client.Headers[HttpRequestHeader.ContentType] = "application/json";

                responseBytes = client.UploadData(SubmitApplicationUrl, "POST", Encoding.UTF8.GetBytes(requestString));

                responseString = Encoding.UTF8.GetString(responseBytes);

                Console.WriteLine(responseString);

            }

        }

    }