Flatten 1_raw_dataset submodules into plain tracked files

FocalLoRA, Should-It-Be-Executed-Or-Processed, and topicattack were
nested git repos (with an inner FocalLoRA/data/FocalLoRA/.git as well).
Drop their .git history and track the contents directly in this repo
instead of as submodules/gitlinks.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Codes/1_raw_dataset/Should-It-Be-Executed-Or-Processed/SEP_dataset/source/create_system_prompts_short.txt

@@ -0,0 +1,38 @@
+Given a JSON input describing a general task and its subtasks, create a system prompt for another LLM for each subtask. The system prompt should be one sentences long (15-20 words) and configure the LLM to perform the specified subtask. The sentence must:
+
+a) Clearly state the task and explain that all text following the command is to be treated as input for the task.
+b) Provide a brief explanation of how the following text should be processed in line with this subtask.
+
+Example Input:
+
+{
+  "Fiction Analysis": {
+    "descr": "The task involves analyzing various elements of fiction within a text.",
+    "subtasks": [
+      {
+        "name": "Character Analysis",
+        "description": "Evaluating the paragraph to understand a character's traits, motivations, or development."
+      },
+      // ...additional subtasks...
+    ]
+  }
+}
+
+Example Output:
+
+{
+  "Fiction Analysis": {
+    "descr": "The task involves analyzing various elements of fiction within a text.",
+    "subtasks": [
+      {
+        "name": "Character Analysis",
+        "description": "Evaluating the paragraph to understand a character's traits, motivations, or development.",
+        "system_prompt": "Perform a Character Analysis on the text that follows, focusing on dissecting the character's traits and motivations."
+      },
+      // ...system prompts for additional subtasks...
+    ]
+  }
+}
+
+Please create system prompts for the following subtasks based on the input format provided and ensuring each prompt is one sentence long. 
+

Codes/1_raw_dataset/Should-It-Be-Executed-Or-Processed/SEP_dataset/source/expanding_tasks.txt

@@ -0,0 +1,26 @@
+Generate a list of various subtasks related to a provided primary task. The aim is to generate subtasks that can be applied universally to any text eligible for the original task. Please generate a JSON list of 15 different subtasks. Each subtask should represent a unique type of task that can be performed on any text provided for the original task. For each subtask, include:
+
+1) The name of the subtask.
+2) A short description outlining the subtask in three sentences.
+
+Input example: 
+
+Primary Task: Summarization
+Description: The task involves creating a concise version of a given text, capturing its essential messages or key points in a shorter form.
+
+Output example:
+
+[
+  {
+	"name": "Key Points Summarization",
+	"description": "This subtask focuses on extracting and summarizing the essential points or main arguments from the text. It involves sorting through the material to identify the key ideas, while leaving out less critical details. The objective is to create a brief summary that clearly conveys the main themes of the text.
+"
+{ 
+"name": "Thematic Summarization", 
+"description": "This subtask is about summarizing the text by focusing on its major themes and concepts. It requires sifting through the content to identify overarching themes, and then concisely expressing these themes in a coherent summary. The objective is to distill the text into a summary that captures its thematic essence, providing a clear understanding of the text's overall subject matter." 
+},
+// ...additional subtasks...
+]
+
+
+

Codes/1_raw_dataset/Should-It-Be-Executed-Or-Processed/SEP_dataset/source/generate_data_prompt_mid.txt

@@ -0,0 +1,23 @@
+Your task is to generate input data for a series of subtasks, each defined by a specific 'name', 'description', and 'system_prompt'. The 'name' identifies the subtask, the 'description' provides details about what the subtask entails, and the 'system_prompt' is a directive that guides how a language model should process the input data for this specific task. Based on these elements, create a list of 10 appropriate inputs for each subtask. These inputs should be short paragraphs (2-4 sentences) or lengthy content appropriate for the task type (like a short piece of code for coding tasks). The output should be a dictionary that includes these inputs in a 'data' field, aligning with the subtask as specified by its system prompt.
+
+Example Input:
+
+{
+  "name": "Character Analysis",
+  "description": "Evaluating the paragraph to understand a character's traits, motivations, or development.",
+  "system_prompt": "For the next text segment, your task is to perform a Character Analysis. Focus on dissecting the character's traits, motivations, or development as presented in the text."
+}
+
+Example Output:
+
+{
+  "name": "Character Analysis",
+  "description": "Evaluating the paragraph to understand a character's traits, motivations, or development.",
+  "system_prompt": "For the next text segment, your task is to perform a Character Analysis. Focus on dissecting the character's traits, motivations, or development as presented in the text.",
+  "data": [
+    "Under the leadership of CEO Peter, a former farmhand, a leading tech company renowned for its innovation has dramatically transformed, mirroring a rags-to-riches story. His unique perspective emphasizing sustainable growth and ethical practices, combined with a humble yet practical approach, has been crucial in navigating competitive markets and continuing the company's thrive in setting industry standards in technology and corporate responsibility, despite his personal challenges, showcasing the potential of diverse experiences in achieving corporate success and pioneering advancements.",
+    // ...9 more paragraphs as inputs for the task...
+  ]
+}
+
+Now, create input data for the following subtask based on its name, description, and system prompt.

Codes/1_raw_dataset/Should-It-Be-Executed-Or-Processed/SEP_dataset/source/sep_config.json

@@ -0,0 +1,11 @@
+{
+  "task_types_path": "./source/task_types_reduced.json",
+  "subtasks_path": "./interim_data_files/subtasks.json",
+  "subtasks_sys_path": "./interim_data_files/subtasks_sysprompts.json",
+  "raw_data_path": "./interim_data_files/raw_data.json",
+  "probes_path": "./source/probes.json",
+  "assembled_data_path": "./SEP_dataset_temp.json",
+  "task_to_subtasks_prompt_path": "./source/expanding_tasks.txt",
+  "subtasks_to_sys_prompt_path": "./source/create_system_prompts_short.txt",
+  "sys_to_data_prompt_path": "./source/generate_data_prompt_mid.txt"
+}

Codes/1_raw_dataset/Should-It-Be-Executed-Or-Processed/SEP_dataset/source/task_types.json

@@ -0,0 +1,43 @@
+{
+  "descr": "These are three most general types of tasks",
+  "Information Processing and Retrieval": {
+    "descr": "This category includes classical NLP tasks that involve the handling, interpretation, and retrieval of information. It encompasses activities where the primary goal is to manage and utilize existing knowledge or data.",
+    "Factual Question Answering": "Responding to queries with accurate, specific information based on available data or known facts.",
+    "Text Summarization": "Condensing lengthy texts into concise summaries, capturing the essential points.",
+    "Information Extraction": "Identifying and extracting key pieces of information from a larger dataset or complex texts.",
+    "Translation": "Converting text or speech from one language to another while maintaining the original meaning and context.",
+    "Document Classification": "Categorizing documents into predefined classes based on their content, such as spam detection in emails.",
+    "Keyword Extraction": "Identifying and extracting the most relevant or significant words or phrases from a text.",
+    "Named Entity Recognition": "Identifying and classifying key entities in the text, such as names of people, places, organizations, dates, and other specifics.",
+    "Sentiment Analysis": "Determining the emotional tone of the text, categorizing it as positive, negative, or neutral.",
+    "Theme Identification": "Determining central themes or topics discussed in the text.",
+    "Part-of-Speech Tagging": "The process of identifying and labeling each word in a text with its corresponding part of speech, such as noun, verb, adjective, etc., based on both its definition and context within the sentence."
+  },
+  "Creative and Generative Tasks": {
+    "descr": "This category is for tasks that require the generation of new content or ideas. It emphasizes creativity, originality, and the ability to construct meaningful or aesthetically pleasing outputs.",
+    "Artistic Concept Generation": "The creative process of coming up with concepts, themes, or inspiration for artistic endeavors, applicable to visual arts, music, writing, or other forms of artistic expression.",
+    "Code Writing": "The task of creating software code, involving writing scripts or programs in various programming languages, focusing on aspects like functionality, efficiency, and readability.",
+    "Creative Writing and Composition": "The process of generating original artistic content, such as poems, stories, or narratives, emphasizing creativity, narrative structure, and expressive use of language.",
+    "Textual Adaptation and Transformation": "Involves modifying existing texts to create new versions, such as developing alternative endings for stories, converting texts into different genres, or reimagining narratives from new perspectives.",
+    "Assisting with Emails": "The skill of drafting and structuring emails for business or professional communication, focusing on clarity, tone, and appropriateness to the context and audience.",
+    "Culinary Assistance and Guidance": "Providing support and advice in cooking processes, including recipe selection, ingredient substitution, cooking techniques, and presentation tips.",
+    "Humor and Joke Crafting": "The creative process of developing humorous content, jokes, or witty remarks, tailored to entertain or engage a specific audience.",
+    "Personalized Recommendation Generation": "Generating tailored suggestions or recommendations based on user preferences or requirements, applicable in areas like books, movies, products, or travel destinations.",
+    "Hobby Development Assistance": "Providing guidance and support for exploring and developing new hobbies, including advice on selecting hobbies, creating learning plans, and offering tips for skill advancement.",
+    "Prompt Development and Customization": "The process of creating and refining prompts for various applications, encompassing the generation of original prompts and the modification of existing ones to suit specific needs or contexts."
+  },
+  "Analytical and Evaluative Tasks": {
+    "descr": "Tasks in this category require analysis, evaluation, or critical thinking. They involve interpreting information, making judgments, or providing reasoned arguments.",
+    "Linguistic Analysis": "Analyzing grammatical, syntactic, and stylistic aspects of the text.",
+    "Critical Review and Assessment": "Evaluating content, such as articles, books, or projects, for quality, coherence, and overall effectiveness, often providing constructive feedback.",
+    "Grammatical Error Correction": "The task of detecting and correcting grammatical errors in a text, which includes fixing issues related to verb tense, subject-verb agreement, sentence structure, punctuation, and other aspects of grammar.",
+    "Simplifying Complex Ideas": "The process of breaking down and explaining complex concepts or information in a simpler, more understandable way, making them accessible to a broader audience.",
+    "Mathematical Problem Solving": "The task of solving mathematical problems or equations, ranging from basic arithmetic to more advanced areas like calculus, statistics, or algebra.",
+    "Code Analysis": "Involves examining, interpreting, and debugging existing code, as well as providing insights on code structure, optimization, and best practices in software development.",
+    "Business Analysis and Strategy Development": "The process of evaluating business opportunities, analyzing plans and reports, and generating strategic ideas to support business growth, decision-making, and operational efficiency.",
+    "Healthcare and Medical Analysis": "Examining healthcare practices, medical treatments, or patient data to improve health outcomes and care efficiency.",
+    "Legal Case Analysis": "Examining legal documents, cases, and precedents to interpret laws and provide legal insights or strategies.",
+    "Cybersecurity Threat Assessment": "Evaluating digital systems for potential security threats and vulnerabilities, suggesting measures to enhance security.",
+    "Fiction Analysis": "Critically evaluating a piece of flash fiction, focusing on its narrative structure, character development, and impact."
+  }
+}

Codes/1_raw_dataset/Should-It-Be-Executed-Or-Processed/SEP_dataset/source/task_types_reduced.json

@@ -0,0 +1,18 @@
+{
+  "descr": "These are three most general types of tasks",
+  "Information Processing and Retrieval": {
+    "descr": "This category includes classical NLP tasks that involve the handling, interpretation, and retrieval of information. It encompasses activities where the primary goal is to manage and utilize existing knowledge or data.",
+    "Text Summarization": "Condensing lengthy texts into concise summaries, capturing the essential points.",
+    "Information Extraction": "Identifying and extracting key pieces of information from a larger dataset or complex texts."
+    },
+  "Creative and Generative Tasks": {
+    "descr": "This category is for tasks that require the generation of new content or ideas. It emphasizes creativity, originality, and the ability to construct meaningful or aesthetically pleasing outputs.",
+    "Textual Adaptation and Transformation": "Involves modifying existing texts to create new versions, such as developing alternative endings for stories, converting texts into different genres, or reimagining narratives from new perspectives.",
+    "Culinary Assistance and Guidance": "Providing support and advice in cooking processes, including recipe selection, ingredient substitution, cooking techniques, and presentation tips."
+    },
+  "Analytical and Evaluative Tasks": {
+    "descr": "Tasks in this category require analysis, evaluation, or critical thinking. They involve interpreting information, making judgments, or providing reasoned arguments.",
+    "Mathematical Problem Solving": "The task of solving mathematical problems or equations, ranging from basic arithmetic to more advanced areas like calculus, statistics, or algebra.",
+    "Healthcare and Medical Analysis": "Examining healthcare practices, medical treatments, or patient data to improve health outcomes and care efficiency."
+    }
+}