ó cÂY]c@sÊdZddlmZddlZddlZddlZddlZddlmZddlm Z m Z m Z ddl m Z mZmZmZmZmZddlZddlmZdd lmZddlZdd lmZed d d dgƒZdd„Zdd„Zddeedddd„Z d„Z!e"d„Z#e"d„Z$e"d„Z%de"d„Z&e"d„Z'e"d„Z(e"d„Z)d„Z*d„Z+d„Z,d „Z-dd!„Z.dS("s Base IO code for all datasets iÿÿÿÿ(tprint_functionN(t namedtuple(tenvirontlistdirtmakedirs(tdirnametexistst expandusertisdirtjointsplitexti(tBunch(tcheck_random_state(t urlretrievetRemoteFileMetadatatfilenameturltchecksumcCsS|dkr*tjdtddƒƒ}nt|ƒ}t|ƒsOt|ƒn|S(ssReturn the path of the scikit-learn data dir. This folder is used by some large dataset loaders to avoid downloading the data several times. By default the data dir is set to a folder named 'scikit_learn_data' in the user home folder. Alternatively, it can be set by the 'SCIKIT_LEARN_DATA' environment variable or programmatically by giving an explicit folder path. The '~' symbol is expanded to the user home folder. If the folder does not already exist, it is automatically created. Parameters ---------- data_home : str | None The path to scikit-learn data dir. tSCIKIT_LEARN_DATAt~tscikit_learn_dataN(tNoneRtgetR RRR(t data_home((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt get_data_homes     cCst|ƒ}tj|ƒdS(s˜Delete all the content of the data home cache. Parameters ---------- data_home : str | None The path to scikit-learn data dir. N(Rtshutiltrmtree(R((s4lib/python2.7/site-packages/sklearn/datasets/base.pytclear_data_home<s tstrictic Cs8g}g} g} gtt|ƒƒD]!} tt|| ƒƒr%| ^q%} |dk r€g| D]} | |kr_| ^q_} nx‹t| ƒD]}\} }| j|ƒt||ƒ}gtt|ƒƒD]}t||ƒ^qÈ}|jt|ƒ| gƒ| j|ƒqWt j | ƒ} t j |ƒ}|rxt |ƒ}t j | j dƒ}|j|ƒ| |} ||}n|rg}x9| D]1}t|dƒ} |j| jƒƒWdQXq‹W|dk rôg|D]}|j||ƒ^qÓ}ntd|d| d| d|d|ƒStd| d| d|d|ƒS( s“Load text files with categories as subfolder names. Individual samples are assumed to be files stored a two levels folder structure such as the following: container_folder/ category_1_folder/ file_1.txt file_2.txt ... file_42.txt category_2_folder/ file_43.txt file_44.txt ... The folder names are used as supervised signal label names. The individual file names are not important. This function does not try to extract features into a numpy array or scipy sparse matrix. In addition, if load_content is false it does not try to load the files in memory. To use text files in a scikit-learn classification or clustering algorithm, you will need to use the `sklearn.feature_extraction.text` module to build a feature extraction transformer that suits your problem. If you set load_content=True, you should also specify the encoding of the text using the 'encoding' parameter. For many modern text files, 'utf-8' will be the correct encoding. If you leave encoding equal to None, then the content will be made of bytes instead of Unicode, and you will not be able to use most functions in `sklearn.feature_extraction.text`. Similar feature extractors should be built for other kind of unstructured data input such as images, audio, video, ... Read more in the :ref:`User Guide `. Parameters ---------- container_path : string or unicode Path to the main folder holding one subfolder per category description : string or unicode, optional (default=None) A paragraph describing the characteristic of the dataset: its source, reference, etc. categories : A collection of strings or None, optional (default=None) If None (default), load all the categories. If not None, list of category names to load (other categories ignored). load_content : boolean, optional (default=True) Whether to load or not the content of the different files. If true a 'data' attribute containing the text information is present in the data structure returned. If not, a filenames attribute gives the path to the files. shuffle : bool, optional (default=True) Whether or not to shuffle the data: might be important for models that make the assumption that the samples are independent and identically distributed (i.i.d.), such as stochastic gradient descent. encoding : string or None (default is None) If None, do not try to decode the content of the files (e.g. for images or other non-text content). If not None, encoding to use to decode text files to Unicode if load_content is True. decode_error : {'strict', 'ignore', 'replace'}, optional Instruction on what to do if a byte sequence is given to analyze that contains characters not of the given `encoding`. Passed as keyword argument 'errors' to bytes.decode. random_state : int, RandomState instance or None (default=0) Determines random number generation for dataset shuffling. Pass an int for reproducible output across multiple function calls. See :term:`Glossary `. Returns ------- data : Bunch Dictionary-like object, the interesting attributes are: either data, the raw text data to learn, or 'filenames', the files holding it, 'target', the classification labels (integer index), 'target_names', the meaning of the labels, and 'DESCR', the full description of the dataset. itrbNtdatat filenamest target_namesttargettDESCR(tsortedRRR Rt enumeratetappendtextendtlentnptarrayR tarangetshapetshuffletopentreadtdecodeR (tcontainer_patht descriptiont categoriest load_contentR,tencodingt decode_errort random_stateR!R Rtftfolderstlabeltfoldert folder_pathtdt documentstindicesRR((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt load_filesHsLY! ( +      (  c Cs tt|d|ƒƒâ}tj|ƒ}t|ƒ}t|dƒ}t|dƒ}tj|dƒ}tj||fƒ}tj|fdtjƒ} xZt |ƒD]L\} } tj | d dtj ƒ|| `. Parameters ---------- return_X_y : boolean, default=False. If True, returns ``(data, target)`` instead of a Bunch object. See below for more information about the `data` and `target` object. Returns ------- data : Bunch Dictionary-like object, the interesting attributes are: 'data', the data to learn, 'target', the classification labels, 'target_names', the meaning of the labels, 'feature_names', the meaning of the features, and 'DESCR', the full description of the dataset. (data, target) : tuple if ``return_X_y`` is True The copy of UCI ML Wine Data Set dataset is downloaded and modified to fit standard format from: https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data Examples -------- Let's say you are interested in the samples 10, 80, and 140, and want to know their class name. >>> from sklearn.datasets import load_wine >>> data = load_wine() >>> data.target[[10, 80, 140]] array([0, 1, 2]) >>> list(data.target_names) ['class_0', 'class_1', 'class_2'] s wine_data.csvtdescrs wine_data.rstNRR!R R"t feature_namestalcoholt malic_acidtashtalcalinity_of_asht magnesiumt total_phenolst flavanoidstnonflavanoid_phenolstproanthocyaninstcolor_intensitythuesod280/od315_of_diluted_winestproline(Rt__file__RQR-R R.R (t return_X_yRHRR!R trst_filetfdescr((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt load_wineûs,2  cCsªttƒ}t|dƒ\}}}t|ddƒ}tt|ddƒƒ}|jƒ}WdQX|rs||fStd|d|d|d|d d d d d gd|ƒS(sLoad and return the iris dataset (classification). The iris dataset is a classic and very easy multi-class classification dataset. ================= ============== Classes 3 Samples per class 50 Samples total 150 Dimensionality 4 Features real, positive ================= ============== Read more in the :ref:`User Guide `. Parameters ---------- return_X_y : boolean, default=False. If True, returns ``(data, target)`` instead of a Bunch object. See below for more information about the `data` and `target` object. .. versionadded:: 0.18 Returns ------- data : Bunch Dictionary-like object, the interesting attributes are: 'data', the data to learn, 'target', the classification labels, 'target_names', the meaning of the labels, 'feature_names', the meaning of the features, 'DESCR', the full description of the dataset, 'filename', the physical location of iris csv dataset (added in version `0.20`). (data, target) : tuple if ``return_X_y`` is True .. versionadded:: 0.18 Notes ----- .. versionchanged:: 0.20 Fixed two wrong data points according to Fisher's paper. The new version is the same as in R, but not as in the UCI Machine Learning Repository. Examples -------- Let's say you are interested in the samples 10, 25, and 50, and want to know their class name. >>> from sklearn.datasets import load_iris >>> data = load_iris() >>> data.target[[10, 25, 50]] array([0, 0, 1]) >>> list(data.target_names) ['setosa', 'versicolor', 'virginica'] siris.csvRRRsiris.rstNR!R R"RSssepal length (cm)ssepal width (cm)spetal length (cm)spetal width (cm)R(RR`RQR R-R.R (RaRHRR!R tiris_csv_filenameRbRc((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt load_irisHs9   c Csttƒ}t|dƒ\}}}t|ddƒ}tt|ddƒƒ}|jƒ}WdQXtjdddd d d d d ddddddddddddddddddd d!d"d#gƒ}|rÜ||fStd|d$|d%|d&|d'|d(|ƒS()sËLoad and return the breast cancer wisconsin dataset (classification). The breast cancer dataset is a classic and very easy binary classification dataset. ================= ============== Classes 2 Samples per class 212(M),357(B) Samples total 569 Dimensionality 30 Features real, positive ================= ============== Read more in the :ref:`User Guide `. Parameters ---------- return_X_y : boolean, default=False If True, returns ``(data, target)`` instead of a Bunch object. See below for more information about the `data` and `target` object. .. versionadded:: 0.18 Returns ------- data : Bunch Dictionary-like object, the interesting attributes are: 'data', the data to learn, 'target', the classification labels, 'target_names', the meaning of the labels, 'feature_names', the meaning of the features, and 'DESCR', the full description of the dataset, 'filename', the physical location of breast cancer csv dataset (added in version `0.20`). (data, target) : tuple if ``return_X_y`` is True .. versionadded:: 0.18 The copy of UCI ML Breast Cancer Wisconsin (Diagnostic) dataset is downloaded from: https://goo.gl/U2Uwz2 Examples -------- Let's say you are interested in the samples 10, 50, and 85, and want to know their class name. >>> from sklearn.datasets import load_breast_cancer >>> data = load_breast_cancer() >>> data.target[[10, 50, 85]] array([0, 1, 0]) >>> list(data.target_names) ['malignant', 'benign'] sbreast_cancer.csvRRRsbreast_cancer.rstNs mean radiuss mean texturesmean perimeters mean areasmean smoothnesssmean compactnesssmean concavitysmean concave pointss mean symmetrysmean fractal dimensions radius errors texture errorsperimeter errors area errorssmoothness errorscompactness errorsconcavity errorsconcave points errorssymmetry errorsfractal dimension errors worst radiuss worst texturesworst perimeters worst areasworst smoothnesssworst compactnesssworst concavitysworst concave pointssworst symmetrysworst fractal dimensionR!R R"RSR( RR`RQR R-R.R(R)R ( RaRHRR!R t csv_filenameRbRcRS((s4lib/python2.7/site-packages/sklearn/datasets/base.pytload_breast_cancer“s66   i c Cs%ttƒ}tjt|ddƒddƒ}tt|ddƒƒ}|jƒ}WdQX|dd…dfjtjƒ}|dd…dd…f}|j ƒ}d|_ |d krç||k} || || }}|| }n|r÷||fSt d|d |d tj d ƒd |d|ƒS(sLoad and return the digits dataset (classification). Each datapoint is a 8x8 image of a digit. ================= ============== Classes 10 Samples per class ~180 Samples total 1797 Dimensionality 64 Features integers 0-16 ================= ============== Read more in the :ref:`User Guide `. Parameters ---------- n_class : integer, between 0 and 10, optional (default=10) The number of classes to return. return_X_y : boolean, default=False. If True, returns ``(data, target)`` instead of a Bunch object. See below for more information about the `data` and `target` object. .. versionadded:: 0.18 Returns ------- data : Bunch Dictionary-like object, the interesting attributes are: 'data', the data to learn, 'images', the images corresponding to each sample, 'target', the classification labels for each sample, 'target_names', the meaning of the labels, and 'DESCR', the full description of the dataset. (data, target) : tuple if ``return_X_y`` is True .. versionadded:: 0.18 This is a copy of the test set of the UCI ML hand-written digits datasets http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits Examples -------- To load the data and visualize the images:: >>> from sklearn.datasets import load_digits >>> digits = load_digits() >>> print(digits.data.shape) (1797, 64) >>> import matplotlib.pyplot as plt #doctest: +SKIP >>> plt.gray() #doctest: +SKIP >>> plt.matshow(digits.images[0]) #doctest: +SKIP >>> plt.show() #doctest: +SKIP Rs digits.csv.gzt delimitert,RRs digits.rstNiÿÿÿÿii R!R timagesR"(iÿÿÿÿii( RR`R(tloadtxtR R-R.tastypeRDtviewR+R R*( tn_classRaRHRR7RRR!t flat_dataRktidx((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt load_digitsês(7  "       c CsÝttƒ}t|dƒ}t|dƒ}tj|ƒ}t|dƒ}tj|ƒ}tt|ddƒƒ}|jƒ}WdQX|r”||fStd|d|d|d d d d d ddddddg d|d|ƒS(s,Load and return the diabetes dataset (regression). ============== ================== Samples total 442 Dimensionality 10 Features real, -.2 < x < .2 Targets integer 25 - 346 ============== ================== Read more in the :ref:`User Guide `. Parameters ---------- return_X_y : boolean, default=False. If True, returns ``(data, target)`` instead of a Bunch object. See below for more information about the `data` and `target` object. .. versionadded:: 0.18 Returns ------- data : Bunch Dictionary-like object, the interesting attributes are: 'data', the data to learn, 'target', the regression target for each sample, 'data_filename', the physical location of diabetes data csv dataset, and 'target_filename', the physical location of diabetes targets csv datataset (added in version `0.20`). (data, target) : tuple if ``return_X_y`` is True .. versionadded:: 0.18 Rsdiabetes_data.csv.gzsdiabetes_target.csv.gzRRs diabetes.rstNR!R"RStagetsextbmitbpts1ts2ts3ts4ts5ts6t data_filenamettarget_filename(RR`R R(RlR-R.R ( RaRHtbase_dirR}RR~R!RbRc((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt load_diabetes:s!   c Cstttƒdƒ}t|dƒ}t|dƒ}tj|ddƒ}tj|ddƒ}t|ƒ}|jƒjƒ}WdQXt|ƒ}|jƒjƒ}WdQXtttƒdƒ}|jƒ} WdQX|ræ||fSt d|d |d |d |d | d |d|ƒS(s Load and return the linnerud dataset (multivariate regression). ============== ============================ Samples total 20 Dimensionality 3 (for both data and target) Features integer Targets integer ============== ============================ Read more in the :ref:`User Guide `. Parameters ---------- return_X_y : boolean, default=False. If True, returns ``(data, target)`` instead of a Bunch object. See below for more information about the `data` and `target` object. .. versionadded:: 0.18 Returns ------- data : Bunch Dictionary-like object, the interesting attributes are: 'data' and 'targets', the two multivariate datasets, with 'data' corresponding to the exercise and 'targets' corresponding to the physiological measurements, as well as 'feature_names' and 'target_names'. In addition, you will also have access to 'data_filename', the physical location of linnerud data csv dataset, and 'target_filename', the physical location of linnerud targets csv datataset (added in version `0.20`). (data, target) : tuple if ``return_X_y`` is True .. versionadded:: 0.18 sdata/slinnerud_exercise.csvslinnerud_physiological.csvtskiprowsiNs/descr/linnerud.rstRRSR!R R"R}R~( R RR`R(RlR-treadlinetsplitR.R ( RaRR}R~t data_exercisetdata_physiologicalR7theader_exercisetheader_physiologicalRR((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt load_linnerudos&$ c Csyttƒ}t|ddƒ}t|ƒ}|jƒ}WdQXt|ddƒ}t|ƒá}tj|ƒ}t|ƒ}t|dƒ}t|dƒ} t j || fƒ} t j |fƒ} t|ƒ}t j |ƒ} xZt |ƒD]L\} }t j |d d t jƒ| | `. Parameters ---------- return_X_y : boolean, default=False. If True, returns ``(data, target)`` instead of a Bunch object. See below for more information about the `data` and `target` object. .. versionadded:: 0.18 Returns ------- data : Bunch Dictionary-like object, the interesting attributes are: 'data', the data to learn, 'target', the regression targets, 'DESCR', the full description of the dataset, and 'filename', the physical location of boston csv dataset (added in version `0.20`). (data, target) : tuple if ``return_X_y`` is True .. versionadded:: 0.18 Notes ----- .. versionchanged:: 0.20 Fixed a wrong data point at [445, 0]. Examples -------- >>> from sklearn.datasets import load_boston >>> boston = load_boston() >>> print(boston.data.shape) (506, 13) RRsboston_house_prices.rstNRsboston_house_prices.csviiiÿÿÿÿR@R!RSR"R(RR`R R-R.RARBRCRDR(RER)R$RFRGR (RaRHt fdescr_nameR7t descr_textRIRKRLRMRNRR!RSROR<((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt load_boston¯s0-    *   cCsÁddlm}tttƒdƒ}tt|dƒƒ}|jƒ}WdQXgtj|ƒD]$}|j dƒr_t||ƒ^q_}g|D]}||ƒ^q}t d|d|d|ƒS( s_Load sample images for image manipulation. Loads both, ``china`` and ``flower``. Read more in the :ref:`User Guide `. Returns ------- data : Bunch Dictionary-like object with the following attributes : 'images', the two sample images, 'filenames', the file names for the images, and 'DESCR' the full description of the dataset. Examples -------- To load the data and visualize the images: >>> from sklearn.datasets import load_sample_images >>> dataset = load_sample_images() #doctest: +SKIP >>> len(dataset.images) #doctest: +SKIP 2 >>> first_img_data = dataset.images[0] #doctest: +SKIP >>> first_img_data.shape #doctest: +SKIP (427, 640, 3) >>> first_img_data.dtype #doctest: +SKIP dtype('uint8') i(timreadRks README.txtNs.jpgRR"( texternals._pilutilRŒR RR`R-R.tosRtendswithR (RŒRHR7RRRRRk((s4lib/python2.7/site-packages/sklearn/datasets/base.pytload_sample_imagesüs$ cCsrtƒ}d}x6t|jƒD]%\}}|j|ƒr|}PqqW|dkrgtd|ƒ‚n|j|S(suLoad the numpy array of a single sample image Read more in the :ref:`User Guide `. Parameters ----------- image_name : {`china.jpg`, `flower.jpg`} The name of the sample image loaded Returns ------- img : 3D array The image as a numpy array: height x width x color Examples --------- >>> from sklearn.datasets import load_sample_image >>> china = load_sample_image('china.jpg') # doctest: +SKIP >>> china.dtype # doctest: +SKIP dtype('uint8') >>> china.shape # doctest: +SKIP (427, 640, 3) >>> flower = load_sample_image('flower.jpg') # doctest: +SKIP >>> flower.dtype # doctest: +SKIP dtype('uint8') >>> flower.shape # doctest: +SKIP (427, 640, 3) sCannot find sample image: %sN(RRR$RRtAttributeErrorRk(t image_nameRktindexROR((s4lib/python2.7/site-packages/sklearn/datasets/base.pytload_sample_image)s  cOsg|jddƒ}t|dƒ\}}tjddkrH||7}n|d ||f}t|ŒS(s†Ensure different filenames for Python 2 and Python 3 pickles An object pickled under Python 3 cannot be loaded under Python 2. An object pickled under Python 2 can sometimes not be loaded correctly under Python 3 because some Python 2 strings are decoded as Python 3 strings which can be problematic for objects that use Python 2 strings as byte buffers for numerical data instead of "real" strings. Therefore, dataset loaders in scikit-learn use different files for pickles manages by Python 2 and Python 3 in the same SCIKIT_LEARN_DATA folder so as to avoid conflicts. args[-1] is expected to be the ".pkl" filename. Under Python 3, a suffix is inserted before the extension to s _pkl_filepath('/path/to/folder', 'filename.pkl') returns: - /path/to/folder/filename.pkl under Python 2 - /path/to/folder/filename_py3.pkl under Python 3+ t py3_suffixt_py3iÿÿÿÿii(RR tsyst version_infoR (targstkwargsR•tbasenametexttnew_args((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt _pkl_filepathRs  cCsgtjƒ}d}t|dƒ:}x0trV|j|ƒ}|sFPn|j|ƒq'WWdQX|jƒS(s.Calculate the sha256 hash of the file at path.i RN(thashlibtsha256R-tTrueR.tupdatet hexdigest(tpatht sha256hasht chunk_sizeR7tbuffer((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt_sha256os  cCsw|dkr|jnt||jƒ}t|j|ƒt|ƒ}|j|krstdj|||jƒƒ‚n|S(sHelper function to download a remote dataset into path Fetch a dataset pointed by remote's url, save into path using remote's filename and ensure its integrity based on the SHA256 Checksum of the downloaded file. Parameters ----------- remote : RemoteFileMetadata Named tuple containing remote dataset meta information: url, filename and checksum dirname : string Directory to save the file to. Returns ------- file_path: string Full path of the created file. sS{} has an SHA256 checksum ({}) differing from expected ({}), file may be corrupted.N( RRR R RR¨RtIOErrortformat(tremoteRt file_pathR((s4lib/python2.7/site-packages/sklearn/datasets/base.pyt _fetch_remote|s  (/t__doc__t __future__RRŽRAR—Rt collectionsRRRRtos.pathRRRRR R RŸtutilsR R tnumpyR(t*sklearn.externals.six.moves.urllib.requestR RRRRR¡R?RQtFalseRdRfRhRrR€RˆR‹RR”RžR¨R­(((s4lib/python2.7/site-packages/sklearn/datasets/base.pytsB    .     ‡ * M K WP 5 @ M - )