{"id":106498,"date":"2025-04-02T16:13:55","date_gmt":"2025-04-02T10:43:55","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/"},"modified":"2025-04-02T15:00:43","modified_gmt":"2025-04-02T09:30:43","slug":"data-science-and-optimization","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/","title":{"rendered":"Data Science And Optimization: Doing The Best With Noise"},"content":{"rendered":"\n<p style=\"background-color: #f4f4f4; padding: 15px; border-left: 5px solid #007bff; font-size: 16px; font-family: Arial, sans-serif; color: #333; margin-bottom: 20px;\">\n  <strong>Author- \n    <a href=\"https:\/\/www.linkedin.com\/in\/kumar-muthuraman-37a2ba5\/\" target=\"_blank\" rel=\"noreferrer noopener\" style=\"color: #007bff; text-decoration: none;\">Prof. Kumar Muthuraman<\/a><br>\n  <\/strong>\n  (The McCombs School of Business at The University of Texas at Austin)\n<br>\n<strong>Edited By- \n    <a href=\"https:\/\/www.linkedin.com\/in\/abhinanda-sarkar-33667b5a\/\" target=\"_blank\" rel=\"noreferrer noopener\" style=\"color: #007bff; text-decoration: none;\">Dr. Abhinanda Sarkar<\/a><br>\n  <\/strong>\n  (Director of Academics, Great Learning)\n<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"1-introduction\"><strong>1. Introduction<\/strong><\/h2>\n\n\n\n<p>A common problem in data science is the estimation of unknown parameters from a sample of data. These parameters can reflect underlying conditions, such as the extent of the relationship between price and sales, or the association between two stocks in a financial portfolio. The estimation of these parameters based on data usually relies on some sort of optimization, i.e. what value of the parameter is closest to the data in some precisely defined sense.&nbsp;<\/p>\n\n\n\n<p>The data can contain information that is irrelevant to the parameter estimation problem. We can think of this as <em>noise,<\/em> as opposed to the relevant information, which is the <em>signal<\/em> in the data. A common approach is to try and separate out the noise and remove it. The signal is then used to obtain an optimal estimate. A well-known example of this is the regression problem, where the parameter can be the slope of a straight line through data in two dimensions. The noise in this case is captured by how far the points are from the straight line, i.e. the size of the errors. Estimation, then, \u2018minimizes the errors\u2019.&nbsp;<\/p>\n\n\n\n<p>But it is quite possible that the noise is actually useful in the estimation problem. This is particularly true if there are many parameters, and the signal is sparse in the information it contains for each parameter. In large data sets, the noise may represent the majority of the data and should be adapted to suitably. This article discusses ways to do this for specific problems. The details are in referenced papers, [1] and [2], and the purpose is to synthesize some learnings that a data scientist will find useful when they do optimal estimation.<\/p>\n\n\n\n\n\n  <style>\n   \n    .section {\n      margin-bottom: 20px;\n\n    }\n    .section-title {\n      font-weight: bold;\n      margin-bottom: 10px;\n    }\n    .math {\n      font-style: italic;\n    }\n    .equation {\n  margin: 10px 0;\n  text-align: center;\n  max-width: 660px;\n  overflow-x: auto;\n}\n\n\/* Add this to help MathJax break long equations *\/\n.MathJax_Display {\n  max-width: 660px !important;\n  overflow-x: auto !important;\n  overflow-y: hidden !important;\n}\n    ul {\n      margin-left: 20px;\n    }\n    li {\n      margin-bottom: 10px;\n    }\n  <\/style>\n  <script src=\"https:\/\/cdnjs.cloudflare.com\/ajax\/libs\/mathjax\/2.7.7\/MathJax.js?config=TeX-MML-AM_CHTML\"><\/script>\n  <script type=\"text\/x-mathjax-config\">\n    MathJax.Hub.Config({\n      tex2jax: {\n        inlineMath: [['$','$'], ['\\\\(','\\\\)']],\n        displayMath: [['$$','$$'], ['\\\\[','\\\\]']],\n        processEscapes: true\n      }\n    });\n  <\/script>\n\n\n<div class=\"section\">\n  <h2 id=\"2-covariance-and-matrices\">2. Covariance and Matrices<\/h2>\n  <p>\n    Consider a dataset of observations. There are $n$ observations, and each observation is a vector of $p$ \n    components. For example, a portfolio of $p$ stocks $(p=5)$ can be observed for 30 days $(n=30)$. For each \n    component, analysts often standardize the data, i.e. subtract a mean and divide a standard deviation. For \n    such a standardized version of the data, we can define a covariance matrix as follows:\n  <\/p>\n\n  <div class=\"equation\">\n    $$V_n = \\frac{1}{n} \\sum_{i=1}^n X_i X_i^T$$\n  <\/div>\n\n  <p>\n    Here $X_i$ is the stock price (in our example) for stock $i$ at time $t$. $V$ is a $p \\times p$ matrix where each element \n    represents the covariance between the $i^{th}$ and $j^{th}$ components. It is a measure of how two components \n    vary with respect to each other. A large positive covariance corresponds to, in our example, two stocks \n    rising and falling together. A large negative covariance corresponds to them moving in opposite \n    directions. In fact, the standardization of the data implies that the covariance is actually a correlation and \n    lies between -1 and +1.\n  <\/p>\n<\/div>\n\n<div class=\"section\">\n  <h2 id=\"3-eigenvalues-signal-and-noise\">3. Eigenvalues, signal, and noise<\/h2>\n  <p>\n    The covariance matrix captures both the signal and the noise in the data. We are interested in the signal \n    and \u2014 often \u2014 we want to remove the noise. One way to capture the signal is to find eigenvalues and \n    eigenvectors. In mathematical terms, this means that we can write the covariance matrix as a \n    decomposition.\n  <\/p>\n\n  <div class=\"equation\">\n    $$V = \\lambda_1 e_1 e_1^T + \\lambda_2 e_2 e_2^T + ... + \\lambda_p e_p e_p^T$$\n  <\/div>\n\n  <p>\n    Here $\\lambda_1 \\geq \\lambda_2 \\geq ... \\geq \\lambda_p$ are eigenvalues arranged in decreasing order. The vectors $e_1, e_2, ..., e_p$ are \n    corresponding eigenvectors. The notation $T$ indicates transposition from a column to a row, so the \n    matrix orders match. In addition, $e_i^T e_i = 1$ and $e_i^T e_j = 0$ for $i \\neq j$. Together the eigenvalues and the \n    corresponding eigenvectors are called an eigendecomposition.\n  <\/p>\n\n  <p>\n    The eigenvalues capture the information in the matrix $V$ in the sense that they capture the variability in \n    the data \u2014 through the covariance matrix. As decreasing values, $(\\lambda_1, e_1)$ captures the most variability \n    and $(\\lambda_p, e_p)$ captures the least variability. As additional condition, $e_i^T e_j = 0$, ensures that the \n    information each eigenpair is distinct (technically, independent or orthogonal) from that of the other \n    eigenpairs.\n  <\/p>\n\n  <p>\n    Simply put (see [1] and [2] for details), we can think of the leading eigenpairs as capturing the signal in \n    the data and the trailing eigenpairs as contributing noise. For example, we can think of $e_1$ as \n    representing the stock weights in a portfolio of stocks that has the most variability and this reflects most \n    clearly what is going on in the broader market. Conversely, $e_p$ represents a portfolio that had the least \n    variability and this captures the non-systemic randomness in the market. Clearly both aspects are \n    important \u2014 the signal to capture risk and the noise to manage it as much as possible. There is also a \n    more technical sense in which the leading eigenpairs contribute to a more statistically stable estimate of \n    the covariance matrix that we will allude to later.\n  <\/p>\n<\/div>\n\n<div class=\"section\">\n  <h2 id=\"4-optimization\">4. Optimization<\/h2>\n  <p>\n    The natural question to then ask would be on how to set up a procedure that best optimizes the mix \n    between the eigenpairs. There are two ways one can distinguish between them in the eigenspairs sense.\n  <\/p>\n\n  <p>\n    In data science, there can be several ways optimization can be used. In [1], optimization is used to find \n    the best financial portfolio, with \"best\" being defined by low risk with high returns (a \"risk-return\" curve). \n    In [2], optimization is used to find the best fit to data, where data include a response variable $(Y)$ that is the \n    function of the components as described $(X)$. In both cases, the problem reduces to finding the optimal fit \n    on eigenpairs associated with the covariance matrix.\n  <\/p>\n\n  <p>\n    For example, consider a version of the financial portfolio optimization problem [1]. Here a possible task \n    could be to find good mix of stocks that would maximize returns while keeping risk, measured by \n    variance:\n  <\/p>\n\n  <ul>\n    <li>The first step is to divide the eigenvalues of the covariance matrix into two sets $\\lambda_1 \\geq \\lambda_2 \\geq ... \\geq \\lambda_k$ \n      and $\\lambda_{k+1} \\geq \\lambda_{k+2} \\geq ... \\geq \\lambda_p$, where the first set of eigenvalues (larger values) corresponds to signal and \n      the second set of second set of eigenvalues (smaller values) corresponds to noise.<\/li>\n      \n    <li>We then need to create two p-dimensional weight vectors $w_s$ and $w_n$, representing the $p$ \n      weights making up the portfolio. Recall that $p$ is the number of stocks in the portfolio.<\/li>\n      \n    <li>We now need to connect the two weight vectors to the corresponding p-dimensional \n      eigenvectors. One way to do this is to force the weights to be orthogonal to the eigenvectors, \n      in slightly cumbersome notation $w_s^T e_i = ... = w_s^T e_k = 0$ (Condition 1) and \n      $w_n^T e_{k+1} = ... = w_n^T e_p = 0$ (Condition 2) thus ensuring that the signal and noise \n      weights fall within the corresponding signal and noise subspaces (i.e., linear combinations \n      are added).<\/li>\n  <\/ul>\n\n  <p>\n    We are now ready to optimize. In matrix notation, using transpose denoted by $T$, we can find optimal \n    weights by:\n  <\/p>\n\n  <div class=\"equation\">\n  $$w_s^* = argmin\\, w_s^T V w_s$$\n  $$\\text{subject to} \\;\\; w_s^T u = 1 \\;\\; \\text{and} \\;\\; w_s \\geq 0$$\n  $$\\text{and} \\;\\; w_s \\;\\; \\text{satisfies Condition 1}$$\n<\/div>\n\n<div class=\"equation\">\n  $$w_n^* = argmin\\, w_n^T V w_n$$\n  $$\\text{subject to} \\;\\; w_n^T u = 1 \\;\\; \\text{and} \\;\\; w_n \\geq 0$$\n  $$\\text{and} \\;\\; w_n \\;\\; \\text{satisfies Condition 2}$$\n<\/div>\n\n  <p>\n    Here $argmin$ is shorthand for 'that which minimizes', $u$ is a vector of ones and, together with the \n    non-negativity condition, makes the weights add up to 1 while disallowing negative weights (in finance, \n    sometimes called the \"no short selling condition\").\n  <\/p>\n\n  <p>\n    Such optimization problems are called quadratic programming problems. In this case, the above \n    optimization problems minimizes the variance \u2014 our risk \u2014 subject to the constraints.\n  <\/p>\n<\/div>\n\n<div class=\"section\">\n  <h2 id=\"5-putting-things-together\">5. Putting things together<\/h2>\n  <p>\n    In the portfolio optimization problem, we have ended up with two optimal portfolios. They are optimal \n    in different ways. The signal-optimizing portfolio is explicitly minimizing risk. The noise-optimizing \n    portfolio is doing something a bit more subtle. It is achieving robustness, in the sense of the portfolio \n    being stable to random, choppy fluctuations. Both of these characteristics are important and it \n    suggests that they should both be criteria in selecting a best portfolio.\n  <\/p>\n\n  <p>\n    More generally, two optimal solutions can be combined. A straightforward way might be to take a linear \n    combination $w^* = \\alpha w_s^* + (1-\\alpha)w_n^*$, that is itself optimal [?]. In such combinations are described, \n    together with appropriate criteria. The problem of the choice of $\\alpha$ can be considered together with the \n    choice of $k$, the number of eigenpairs to consider as signal \u2014 a question we had left unanswered.\n  <\/p>\n<\/div>\n\n\n\n\n\n<p><em>[1] Long Zhao, Deepayan Chakrabarti, and Kumar Muthuraman, Portfolio Construction by Mitigating Error Amplification: The Bounded-Noise Portfolio<\/em><br><em><br>[2] Long Zhao, Deepayan Chakrabarti, and Kumar Muthuraman, Unified Classical and Robust Optimization for Least Squares<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Data science faces the challenge of estimating unknown parameters from noisy data. This article explains how optimizing the balance between signal and noise using techniques like eigenvalues and eigenvectors leads to better parameter estimates, especially in complex problems like financial portfolio optimization.<\/p>\n","protected":false},"author":41,"featured_media":106522,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_uag_custom_page_level_css":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[9],"tags":[36860],"content_type":[],"class_list":["post-106498","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science","tag-marketing-team"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.3 (Yoast SEO v27.3) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Data Science And Optimization: Doing The Best With Noise<\/title>\n<meta name=\"description\" content=\"Learn how data science and optimization help improve parameter estimation by separating signal from noise and making optimal decisions in complex problems.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Data Science And Optimization: Doing The Best With Noise\" \/>\n<meta property=\"og:description\" content=\"Learn how data science and optimization help improve parameter estimation by separating signal from noise and making optimal decisions in complex problems.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/\" \/>\n<meta property=\"og:site_name\" content=\"Great Learning Blog: Free Resources what Matters to shape your Career!\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/GreatLearningOfficial\/\" \/>\n<meta property=\"article:published_time\" content=\"2025-04-02T10:43:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1536\" \/>\n\t<meta property=\"og:image:height\" content=\"1024\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"Great Learning Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@https:\/\/twitter.com\/Great_Learning\" \/>\n<meta name=\"twitter:site\" content=\"@Great_Learning\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Great Learning Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/\"},\"author\":{\"name\":\"Great Learning Editorial Team\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/person\\\/6f993d1be4c584a335951e836f2656ad\"},\"headline\":\"Data Science And Optimization: Doing The Best With Noise\",\"datePublished\":\"2025-04-02T10:43:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/\"},\"wordCount\":1470,\"publisher\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/04\\\/data-science-optimizations.webp\",\"keywords\":[\"marketing team\"],\"articleSection\":[\"Data Science and Analytics\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/\",\"name\":\"Data Science And Optimization: Doing The Best With Noise\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/04\\\/data-science-optimizations.webp\",\"datePublished\":\"2025-04-02T10:43:55+00:00\",\"description\":\"Learn how data science and optimization help improve parameter estimation by separating signal from noise and making optimal decisions in complex problems.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/04\\\/data-science-optimizations.webp\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/04\\\/data-science-optimizations.webp\",\"width\":1536,\"height\":1024,\"caption\":\"Data science and Optimizations\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science-and-optimization\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog\",\"item\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Data Science and Analytics\",\"item\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Data Science And Optimization: Doing The Best With Noise\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\",\"name\":\"Great Learning Blog\",\"description\":\"Learn, Upskill &amp; Career Development Guide and Resources\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\"},\"alternateName\":\"Great Learning\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\",\"name\":\"Great Learning\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/GL-Logo.jpg\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/GL-Logo.jpg\",\"width\":900,\"height\":900,\"caption\":\"Great Learning\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/GreatLearningOfficial\\\/\",\"https:\\\/\\\/x.com\\\/Great_Learning\",\"https:\\\/\\\/www.instagram.com\\\/greatlearningofficial\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/school\\\/great-learning\\\/\",\"https:\\\/\\\/in.pinterest.com\\\/greatlearning12\\\/\",\"https:\\\/\\\/www.youtube.com\\\/user\\\/beaconelearning\\\/\"],\"description\":\"Great Learning is a leading global ed-tech company for professional training and higher education. It offers comprehensive, industry-relevant, hands-on learning programs across various business, technology, and interdisciplinary domains driving the digital economy. These programs are developed and offered in collaboration with the world's foremost academic institutions.\",\"email\":\"info@mygreatlearning.com\",\"legalName\":\"Great Learning Education Services Pvt. Ltd\",\"foundingDate\":\"2013-11-29\",\"numberOfEmployees\":{\"@type\":\"QuantitativeValue\",\"minValue\":\"1001\",\"maxValue\":\"5000\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/person\\\/6f993d1be4c584a335951e836f2656ad\",\"name\":\"Great Learning Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"caption\":\"Great Learning Editorial Team\"},\"description\":\"The Great Learning Editorial Staff includes a dynamic team of subject matter experts, instructors, and education professionals who combine their deep industry knowledge with innovative teaching methods. Their mission is to provide learners with the skills and insights needed to excel in their careers, whether through upskilling, reskilling, or transitioning into new fields.\",\"sameAs\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/\",\"https:\\\/\\\/in.linkedin.com\\\/school\\\/great-learning\\\/\",\"https:\\\/\\\/x.com\\\/https:\\\/\\\/twitter.com\\\/Great_Learning\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCObs0kLIrDjX2LLSybqNaEA\"],\"award\":[\"Best EdTech Company of the Year 2024\",\"Education Economictimes Outstanding Education\\\/Edtech Solution Provider of the Year 2024\",\"Leading E-learning Platform 2024\"],\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/author\\\/greatlearning\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Data Science And Optimization: Doing The Best With Noise","description":"Learn how data science and optimization help improve parameter estimation by separating signal from noise and making optimal decisions in complex problems.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/","og_locale":"en_US","og_type":"article","og_title":"Data Science And Optimization: Doing The Best With Noise","og_description":"Learn how data science and optimization help improve parameter estimation by separating signal from noise and making optimal decisions in complex problems.","og_url":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/","og_site_name":"Great Learning Blog: Free Resources what Matters to shape your Career!","article_publisher":"https:\/\/www.facebook.com\/GreatLearningOfficial\/","article_published_time":"2025-04-02T10:43:55+00:00","og_image":[{"width":1536,"height":1024,"url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations.webp","type":"image\/webp"}],"author":"Great Learning Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/twitter.com\/Great_Learning","twitter_site":"@Great_Learning","twitter_misc":{"Written by":"Great Learning Editorial Team","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/#article","isPartOf":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/"},"author":{"name":"Great Learning Editorial Team","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/person\/6f993d1be4c584a335951e836f2656ad"},"headline":"Data Science And Optimization: Doing The Best With Noise","datePublished":"2025-04-02T10:43:55+00:00","mainEntityOfPage":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/"},"wordCount":1470,"publisher":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/#primaryimage"},"thumbnailUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations.webp","keywords":["marketing team"],"articleSection":["Data Science and Analytics"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/","url":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/","name":"Data Science And Optimization: Doing The Best With Noise","isPartOf":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/#primaryimage"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/#primaryimage"},"thumbnailUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations.webp","datePublished":"2025-04-02T10:43:55+00:00","description":"Learn how data science and optimization help improve parameter estimation by separating signal from noise and making optimal decisions in complex problems.","breadcrumb":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/#primaryimage","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations.webp","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations.webp","width":1536,"height":1024,"caption":"Data science and Optimizations"},{"@type":"BreadcrumbList","@id":"https:\/\/www.mygreatlearning.com\/blog\/data-science-and-optimization\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog","item":"https:\/\/www.mygreatlearning.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Data Science and Analytics","item":"https:\/\/www.mygreatlearning.com\/blog\/data-science\/"},{"@type":"ListItem","position":3,"name":"Data Science And Optimization: Doing The Best With Noise"}]},{"@type":"WebSite","@id":"https:\/\/www.mygreatlearning.com\/blog\/#website","url":"https:\/\/www.mygreatlearning.com\/blog\/","name":"Great Learning Blog","description":"Learn, Upskill &amp; Career Development Guide and Resources","publisher":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization"},"alternateName":"Great Learning","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.mygreatlearning.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization","name":"Great Learning","url":"https:\/\/www.mygreatlearning.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/06\/GL-Logo.jpg","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/06\/GL-Logo.jpg","width":900,"height":900,"caption":"Great Learning"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/GreatLearningOfficial\/","https:\/\/x.com\/Great_Learning","https:\/\/www.instagram.com\/greatlearningofficial\/","https:\/\/www.linkedin.com\/school\/great-learning\/","https:\/\/in.pinterest.com\/greatlearning12\/","https:\/\/www.youtube.com\/user\/beaconelearning\/"],"description":"Great Learning is a leading global ed-tech company for professional training and higher education. It offers comprehensive, industry-relevant, hands-on learning programs across various business, technology, and interdisciplinary domains driving the digital economy. These programs are developed and offered in collaboration with the world's foremost academic institutions.","email":"info@mygreatlearning.com","legalName":"Great Learning Education Services Pvt. Ltd","foundingDate":"2013-11-29","numberOfEmployees":{"@type":"QuantitativeValue","minValue":"1001","maxValue":"5000"}},{"@type":"Person","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/person\/6f993d1be4c584a335951e836f2656ad","name":"Great Learning Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","caption":"Great Learning Editorial Team"},"description":"The Great Learning Editorial Staff includes a dynamic team of subject matter experts, instructors, and education professionals who combine their deep industry knowledge with innovative teaching methods. Their mission is to provide learners with the skills and insights needed to excel in their careers, whether through upskilling, reskilling, or transitioning into new fields.","sameAs":["https:\/\/www.mygreatlearning.com\/","https:\/\/in.linkedin.com\/school\/great-learning\/","https:\/\/x.com\/https:\/\/twitter.com\/Great_Learning","https:\/\/www.youtube.com\/channel\/UCObs0kLIrDjX2LLSybqNaEA"],"award":["Best EdTech Company of the Year 2024","Education Economictimes Outstanding Education\/Edtech Solution Provider of the Year 2024","Leading E-learning Platform 2024"],"url":"https:\/\/www.mygreatlearning.com\/blog\/author\/greatlearning\/"}]}},"uagb_featured_image_src":{"full":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations.webp",1536,1024,false],"thumbnail":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations-150x150.webp",150,150,true],"medium":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations-300x200.webp",300,200,true],"medium_large":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations-768x512.webp",768,512,true],"large":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations-1024x683.webp",1024,683,true],"1536x1536":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations.webp",1536,1024,false],"2048x2048":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations.webp",1536,1024,false],"web-stories-poster-portrait":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations-640x853.webp",640,853,true],"web-stories-publisher-logo":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations-96x96.webp",96,96,true],"web-stories-thumbnail":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2025\/04\/data-science-optimizations-150x100.webp",150,100,true]},"uagb_author_info":{"display_name":"Great Learning Editorial Team","author_link":"https:\/\/www.mygreatlearning.com\/blog\/author\/greatlearning\/"},"uagb_comment_info":0,"uagb_excerpt":"Data science faces the challenge of estimating unknown parameters from noisy data. This article explains how optimizing the balance between signal and noise using techniques like eigenvalues and eigenvectors leads to better parameter estimates, especially in complex problems like financial portfolio optimization.","_links":{"self":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/106498","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/users\/41"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/comments?post=106498"}],"version-history":[{"count":26,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/106498\/revisions"}],"predecessor-version":[{"id":108041,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/106498\/revisions\/108041"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/media\/106522"}],"wp:attachment":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/media?parent=106498"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/categories?post=106498"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/tags?post=106498"},{"taxonomy":"content_type","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/content_type?post=106498"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}