­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ 3 njWa.@s4dZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z mZmZmZddlmZddlmZmZmZdZdZdZd Zd Zd Zd Zd ZdZdZ dZ!dZ"Gddde j#Z$Gddde%Z&ddZ'ddZ(d d!Z)d"d#Z*Gd$d%d%e%Z+d&d'Z,d(d)Z-dd+d,Z.d-d.Z/ej0d/d0Z1d1d2Z2d3d4Z3d5d6Z4d7d8Z5d9d:Z6d;d<Z7d=d>Z8d?d@Z9dAdBZ:dCdDZ;dEdFZdKdLZ?dMdNZ@e+dOe/e+dPe.dQgdRdSe+dTe1e+dUe2e+dVe3e+dWe.dWge+dXe4e+dYe,dZe+d[e,d\e+d]e,d^e+d_e,d`e+dae,dbe+dce,dde+dee,dfe+dge,dhe+die,dje+dke-dldme+dne5e+doe-dpdqe+dre6e+dse7e+dte8gZAe'dudvjBee'dwdwe'dxdyjBee'dzd{e(d|d}e'd~de)dde)dde)dde)dde)dde)dde)dde)dde)dde(dde)dde(dee)dde'dde'dde'dde'dde*de9e'dde(dde*de;e'dde'dde*de<e(dde'dde*de=e'ddjBee(dee(dde*de>e*de?e'dde'dde'dde*de@e'dde'ddg,ZCejDe jEddd„ZFdddĄZGdS)aODoctor v2 vitals collector (KPT-5984). Python port of the kcdoctor.sh (1.0-8) vitals collection: gathers the same data set into a DataPackage-based archive processed by the patch server generic uploads pipeline. Scalar and short values are consolidated into a single ``doctor.json`` (the eportal doctor precedent), one field each, instead of a tiny archive entry per value; bulk data (logs, command dumps, package lists, sysfs tunables, crash artifacts) stays as separate archive entries. The 1:1 inventory of collected items lives in docs/features/KPT-5984-doctor-v2/parity-inventory.md. N) authconfigconfig_handlers delivery_kit http_utilskcare log_utilsplatform_utilsserveridutils) run_command) HTTPErrorURLErrorhttplibz2.0-1z/boot/grub2/grub.cfgz/etc/kdump.confz/var/log/dpkg.logz/etc/apt/sources.list*z/etc/yum.repos.d/*z/var/log/libcare/*.logi'z/sysenable*nr_*max_**cnt* /sys/kernel/debug/tracing/events/sys/kernel/tracing/events /sys/devices [0-9]*.logkmsg*logc@s eZdZdZdZeddZdS)KcarectlDoctorPackagezkcarectl-doctorz/upload/kcarectl-doctor/cCstjS)N)rDOCTOR_REPORT_MAX_SIZE_BYTES)selfr,/usr/libexec/kcare/python/kcarectl/doctor.pymax_sizeFszKcarectlDoctorPackage.max_sizeN)__name__ __module__ __qualname__ data_type upload_uripropertyrrrrrrBsrc@s eZdZdZddZddZdS)CollectionSpeczA single bulk vitals item ported from kcdoctor.sh. Collection failures are recorded in errors.log and never abort the report: an unhandled exception inside the package context manager would remove the whole archive. cCs||_||_dS)N)arcname_collect)rr&collectrrr__init__WszCollectionSpec.__init__cCsNy|j||jWn6tk rH}z|jdj|j|WYdd}~XnXdS)Nzfailed to collect {0}: {1})r'r& Exception log_errorformat)r data_packageerrrapply\szCollectionSpec.applyN)rr r!__doc__r)r/rrrrr%Osr%csfdd}t||S)z4`run ""` kcdoctor.sh helper (no shell features)cs|j|dS)N) add_stdout)r-r&)cmdrrr(hszrun_spec..collect)r%)r&r2r(r)r2rrun_specds r3csfdd}t||S)z4`dump ""` kcdoctor.sh helper for regular filescs|j|ddS)N)src_path)add_file)r-r&)pathrrr(sszdump_spec..collect)r%)r&r6r(r)r6r dump_specos r7csfdd}t||S)z`dump ""` for /proc and /sys files. tar reports them as 0-size so they are first read into memory (see the same approach in anomaly.prepare_kernel_anomaly_report). c sLtjjs |jdjdStd}|j||jdWdQRXdS)Nzfile not found: {0}rb) data_bytes)osr6existsr+r,openr5read)r-r&f)r6rrr(s   zproc_dump_spec..collect)r%)r&r6r(r)r6rproc_dump_speczs r?cCs t||S)N)r%)r&fnrrr callable_specsrAc@s eZdZdZddZddZdS) FieldSpeczA single doctor.json field ported from kcdoctor.sh. Collection failures are recorded in errors.log and never abort the report; a collector returning None omits the field (e.g. a distro-specific source file absent on this system). cCs||_||_dS)N)keyr')rrCr(rrrr)szFieldSpec.__init__cCsXy |j}Wn4tk r@}z|jdj|j|dSd}~XnX|dk rT|||j<dS)Nzfailed to collect {0}: {1})r'r*r+r,rC)rr-reportvaluer.rrrr/s zFieldSpec.applyN)rr r!r0r)r/rrrrrBsrBcsfdd}|S)z`dump ""` as a doctor.json string field. Read in binary (non-utf8 safe) and decoded leniently; an absent distro-specific source file simply omits the field (None). c s:tjjsdStd}|jjddjSQRXdS)Nr8zutf-8replace)r:r6r;r<r=decodestrip)r>)r6rrr(s  zread_field..collectr)r6r(r)r6r read_fields rIcs"tjtj|fdd}|S)z`grep ` as a doctor.json list-of-lines field. Match on bytes (like collect_apt_sources) so a non-utf8 byte cannot lose a line; the matched lines are decoded leniently for the JSON. c sDtjjsdStd}|jj}WdQRXfdd|DS)Nr8cs"g|]}j|r|jddqS)zutf-8rF)searchrG).0line)regexrr sz/grep_field..collect..)r:r6r;r<r= splitlines)r>lines)r6rMrrr(s   zgrep_field..collect)recompiler bstr)r6patternr(r)r6rMr grep_fields rUFcsfdd}|S)z@`run ""` captured as a doctor.json field instead of a file.cs0tddd\}}}|pdj}r,|jS|S)NT) catch_stdout catch_stderr)r rHrO)_stdouttext)argvas_linesrrr(s zcmd_field..collectr)r\r]r(r)r\r]r cmd_fieldsr^cCstS)N)DOCTOR_VERSIONrrrrcollect_doctor_versionsr`c Cs<y"tjtjd}tj|jjStk r6dSXdS)zthe public IP of the machine as seen by the patch server. Cached: the main_ip and server_id fields share one HTTP roundtrip per run. ZmyipZNAN)rurlopenr get_patch_server_urlnstrr=rHr*)responserrr get_main_ips recCs$tj}|stjdd}tj|S)N.rY)r get_serveridrerFr rc) server_idrrrcollect_server_idsricCstjS)N)rget_kernel_hashrrrrcollect_kernel_idsrkcCsBi}x8d D]0}tdd|gddd \}}}|p0d j||<q W|S) Narmpouname-T)rVrWrX)rlrmrnrorp)r rH)resultflagrYrZrrr collect_uname s  ruc CsHtjjtsdSttd}|jjdd}WdQRXdd|jDS)z7`grep vmlinuz /boot/grub2/grub.cfg | sed 's/root=.*//'`Nr8zutf-8rFcSs"g|]}d|krtjdd|qS)Zvmlinuzzroot=.*rX)rQsub)rKrLrrrrNsz)collect_grub2_entries..)r:r6r; GRUB2_CFGr<r=rGrO)r>r[rrrcollect_grub2_entriess   rxcCsttjtS)N)sortedglobYUM_REPOS_GLOBrrrrcollect_yum_repos sr|cCsddtjD}|tjdS)z8presence-only port of the kcdoctor.sh detect_cp() probescSsg|]\}}|r|qSrr)rKnameproberrrrN)sz)collect_control_panel..)cpZ softaculous)r CONTROL_PANEL_PROBEShas_softaculous)Zpanelsrrrcollect_control_panel&srcCsttjp dS)zthe kcare.conf options kcarectl refuses to run on, named with the reason. --doctor is exempt from that refusal so a stuck host can still be diagnosed; the field is omitted (None) on a usable config. N)dictrINVALID_CONFIG_OPTIONSrrrrcollect_invalid_config_options-srcCs@tddgddd\}}}|j|tj|p(djdddd d d S) z`ipcs -m | sed -e s/-/=/g`ipcsz-mT)rVrWrXrr=zutf-8)encoding)r9N)r r5r rSrF)r-r&rYrZrrr collect_ipcs:src Cshtjjts |jdjtdSttd}dd|jjD}WdQRX|j |dj |dddS)z$`grep ' install ' /var/log/dpkg.log`zfile not found: {0}Nr8cSsg|]}d|kr|qS)s install r)rKrLrrrrNIsz,collect_dpkg_install_log.. )r9) r:r6r;DPKG_LOGr+r,r<r=rOr5join)r-r&r>rPrrrcollect_dpkg_install_logAs    rcCstjjdstjjdrd}nXtjjdrrd}yt|dWqvtk rn}z|jdj|WYdd}~XqvXnd}|j||dS) Nz /usr/bin/rpmz/bin/rpmzFrpm -q -a --queryformat="%{N}|%{V}-%{R}|%{arch}|%{INSTALLTIME:date}\n"z /usr/bin/dpkgzJ/usr/bin/dpkg-query -W -f "${binary:Package}|${Version}|${Architecture}\n"zdpkg.logzfailed to collect dpkg.log: {0}zecho "unknown package manager")r:r6r;rr*r+r,r1)r-r& packages_cmdr.rrrcollect_packagesNs $rc sFtjdg}xdttjtD]R}tjj|rfx@tj|D]&\}}|j fddt|Dq:Wq|j |qW|s|j dj tdSg}x|D]}y"t |d}|j}WdQRXWn6tk r} z|j dj || wWYdd} ~ XnXtjdj |d d |j fd d|jDqW|j|d j|d d dS)z/`grep -rE '^(deb|URIs)' /etc/apt/sources.list*`s ^(deb|URIs)c3s|]}tjj|VqdS)N)r:r6r)rKr})rootrr jsz&collect_apt_sources..zfile not found: {0}Nr8zfailed to read {0}: {1}z{0}:zutf-8)rc3s |]}j|r|VqdS)N)rJ)rKrL)prefixrMrrrsr)r9)rQrRryrzAPT_SOURCES_GLOBr:r6isdirwalkextendappendr+r,r<r=r*r rSrOr5r) r-r&filesr6rYnamesrPr>contentr.r)rrMrrcollect_apt_sourcesas*  "  "rc Csttjt}|s&|jdjtdSg}x|D]}y0t|d}|jjt d}WdQRXWn6t k r}z|jdj||w0WYdd}~XnXt j dj|dd}|j |dj |q0W|j|dj |dd dS) z%`tail -n10000 /var/log/libcare/*.log`zfile not found: {0}Nr8zfailed to read {0}: {1}z ==> {0} <== zutf-8)rr)r9)ryrzLIBCARE_LOGS_GLOBr+r,r<r=rOLOG_TAIL_LINESr*r rSrrr5) r-r&pathschunksr6r>rPr.headerrrrcollect_libcare_logss  $rcstjjtjsdS|j|djtjxttjtjD]d\}}}xXt|D]Lt fddt DrHtjj |}tjj |tj}|j dj||dqHWq4WdS)z3crashreporter artifacts from /var/cache/kcare/dumpsNz ls -lR {0}c3s|]}tj|VqdS)N)fnmatch)rKrT)r}rrrsz&collect_crash_dumps..zcrashreporter/{0})r4)r:r6rr KDUMPS_DIRr1r,rryanyCRASH_DUMP_PATTERNSrrelpathr5)r-r&rrYrr6Zrel_pathr)r}rcollect_crash_dumpssrcCstjjtsdStj}tjj|s(dS|j|dj|xTt t j tjj |dD]8}tjj|sfqT|j djtjj |tjj |ddqTWdS)Nz ls -lR {0}*zkdump/{0}/vmcore-dmesg.txtzvmcore-dmesg.txt)r4)r:r6r; KDUMP_CONFrget_kdump_rootrr1r,ryrzrr5basename)r-r& kdump_rootZevtrrr collect_kdumps   rc sg}xtjtD]\}}}|jtr2g|dd<qxt|D]tfddtDr.r8zcat {0} r)r9)r:r SYSFS_ROOT startswithSYSFS_EXCLUDE_PREFIXESryrSYSFS_NAME_PATTERNSr6rr<r=r*rr rSr,r5) r-r&rrdirsrr6r>rr)r}rcollect_sysfs_configs    &rZdoctor_versionZ virt_whatz/usr/libexec/kcare/virt-whatT)r]Zmain_iprhZ kernel_iddaterqZredhat_releasez/etc/redhat-releaseZdebian_versionz/etc/debian_versionZ os_releasez/etc/os-releaseZissuez /etc/issueZsysconfig_kernelz/etc/sysconfig/kernelZ proc_uptimez /proc/uptimeZ proc_loadavgz /proc/loadavgZ proc_cmdlinez /proc/cmdline proc_versionz /proc/versionZ default_grubz/etc/default/grubDEFAULTZ grub2_entriesZ sshd_portz/etc/ssh/sshd_configZPortZ yum_repos_dZ control_panelZinvalid_config_optionssyslogztail -n{0} /var/log/syslogdmesgmessagesztail -n{0} /var/log/messagesls_var_cache_kcarezls -lR /var/cache/kcare/z kcare.confz/etc/sysconfig/kcare/kcare.confcpuinfozcat /proc/cpuinfoZ proc_vmstatz /proc/vmstatZ proc_devicesz /proc/devicesZproc_diskstatsz/proc/diskstatsZ proc_mdstatz /proc/mdstatZ proc_meminfoz /proc/meminfoZ proc_swapsz /proc/swapsZproc_filesystemsz/proc/filesystemsZ proc_mountsz /proc/mountsZproc_interruptsz/proc/interruptsz grub.confz/boot/grub/grub.conf proc_modulesz /proc/modulesz grub2.cfgZ proc_zoneinfoz/proc/zoneinfoZls_boot_configsz:ls /etc/grub.conf /boot/grub/grub.conf /boot/grub/menu.lstZls_bootz ls -l /bootZprintenvZ dmidecodersysctlz sysctl -az sysctl.confz/etc/sysctl.confz packages.listZlspciz lspci -vvZdpkg_lzdpkg -lZ apt_sourceszyum.confz /etc/yum.confZ yum_repolistz yum repolistZ libcare_logsz kcarectl.logz tail -n{0} /var/log/kcarectl.logz kdump.confz kcare-cronz/etc/cron.d/kcare-cronzcrashreporter/lszkdump/lsZ aa_statusz aa-statusZsestatusZ kprobes_listz"cat /sys/kernel/debug/kprobes/listZ sysfs_configZlsblkzlsblk -fZdfzdf -h)loggerc Cs\t}|Hi}xtD]}|j||qW|jd|xtD]}|j|q= 400, so a redirect is not treated as a failure; the proxy and patch server never redirect uploads. :param fallback: legacy ``kcdoctor.sh`` runner, injected by the caller (KPT-6065) to keep this module free of an ``__init__`` import cycle. :param force_fallback: when True (``--doctor --fallback``), skip the v2 path and run ``fallback`` directly; the per-invocation form of ``config.FORCE_DOCTOR_FALLBACK``. z--fallback requestedzFORCE_DOCTOR_FALLBACK is setz.doctor: {0}; using the legacy kcdoctor.sh flowF) print_msgNzAdoctor: host is not registered; using the legacy kcdoctor.sh flowzCdoctor: report collection failed; using the legacy kcdoctor.sh flowzdoctor: report uploaded as {0}z,Please provide this filename to support: {0}zJdoctor: report upload unavailable ({0}); using the legacy kcdoctor.sh flow)rFORCE_DOCTOR_FALLBACKrloginfor,rget_http_auth_stringrlogwarnsendr print_wrapperr rr HTTPExceptionsocketerrorremove_archive)fallbackforce_fallbacksrcr- upload_nameerrrrrsend_doctor_reportDs4)    r)rrrr)rrr)rr)F)F)Hr0rrzr:rQrkcarectlrrrrrrrr r r Zkcarectl.process_utilsr Z kcarectl.py23r rrr_rwrrrr{rrrrrr DataPackagerobjectr%r3r7r?rArBrIrUr^r`cachedrerirkrurxr|rrrrrrrrrrrr,r catch_errorsrrrrrrrs0          #